Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Zameer Manji]

It isn't issue if it a broadening of possibilities.

I would like to point out that, if we do want to have a better contract in
the future (ie a set of CommandInfos), we are setting up ourselves for a
API change in the future.

If we clearly document that the CommandInfo is just passed down to the
executor by the master and agent, I don't see any harm.


On Fri, Nov 4, 2016 at 6:53 AM, Joris Van Remoortere 
wrote:

> @zameer
>
> I think your example makes a lot of sense. I didn't interpret the proposal
> as one that would prevent your case.
>
> The way I read the proposal is that we want to allow setting both, not
> require it. If I misunderstood then please ignore my comments.
>
> When I first heard about this proposal it seemed like a nice way for
> frameworks and executors to start defining a more structured contract _if
> they wanted to_, while still allowing others to keep passing their (from
> Mesos's view) unstructured data as long as both sides agree on the
> serialization / de-serialization.
>
> Does this still seem like an issue if it's not a requirement but just a
> broadening of possibilities?
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Wed, Nov 2, 2016 at 1:49 PM, Zameer Manji  wrote:
>
> > Joris,
> >
> > You make a good point. However, I'm not convinced that `CommandInfo`
> should
> > be the well defined construct that people use. Can you please describe
> > different custom executors, and the overlap between them and how
> > CommandInfo will reduce that overlap? I'm having a hard time seeing where
> > CommandInfo will solve all of their cases.
> >
> > Consider the cause of Thermos (Aurora's Executor), it could never use a
> > `CommandInfo` struct because it executes a processes graph instead of a
> > single command.
> >
> > If the project wants to go down this path, I think generalizing
> > `CommandInfo` that could capture more cases (ie multiple commands or a
> > graph of commands) would be a better first step.
> >
> > What do you think?
> >
> > On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> > jo...@mesosphere.io>
> > wrote:
> >
> > > I do think it would be valuable to have a more well defined contract
> > > between frameworks and custom executors.
> > >
> > > As Zameer pointed out a specific framework and accompanying custom
> > executor
> > > can decide to do that in the data bytes; however, if we started
> building
> > > out a few different flavors of executors then it would be great for
> there
> > > to be standard way to pass command information to them.
> > >
> > > The current model works well in a 1-1 mapping between framework and
> > > executor binaries. In a world where that is 1-N it means all N
> executors
> > > have to use the same method of passing the command.
> > >
> > > —
> > > *Joris Van Remoortere*
> > > Mesosphere
> > >
> > > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji 
> wrote:
> > >
> > > > I'm not convinced this is a valid use case.
> > > >
> > > > Mesos is supposed to be a generic kernel for launching "tasks",
> > whatever
> > > > they might be.
> > > >
> > > > In some cases it is useful to launch an executable, in other cases it
> > > might
> > > > be useful to launch a series of executables, and in some other cases
> it
> > > > might be useful to spawn a thread to do some work. Whatever that
> might
> > > be,
> > > > it doesn't matter to Mesos and the executor and framework are free to
> > > > establish a contract in `ExecutorInfo.data`, completely independent
> of
> > > the
> > > > Mesos API.
> > > >
> > > > I think formalizing this contract between executors and frameworks
> via
> > > > CommandInfo is going to introduce more problems than what they solve.
> > If
> > > > the CommandInfo struct is useful, frameworks and executors can just
> > stuff
> > > > that into ExecutorInfo.data, however it's not something that they
> need
> > to
> > > > adhere too.
> > > >
> > > > What's the underlying motivation for this?
> > > >
> > > >
> > > >
> > > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent 
> wrote:
> > > >
> > > > > For command task, if its `ExecutorInfo` would set with
> > > `CommandExecutor`
> > > > as
> > > > > well?
> > > > >
> > > > > Some tickets may relate to this.
> > > > >
> > > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > > >
> > > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are contemplating whether to allow both CommandInfo and
> > > ExecutorInfo
> > > > > on
> > > > > > TaskInfo (MESOS-6294  > > jira/browse/MESOS-6294
> > > > > >).
> > > > > > Currently we only allow one or the other. The motivation is to
> > allow
> > > > > custom
> > > > > > executors a more structured way to pass information (e.g,
> command)
> > > > about
> > > > > > Task. Right now custom executors have to get this data v

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Joris Van Remoortere]

@zameer

I think your example makes a lot of sense. I didn't interpret the proposal
as one that would prevent your case.

The way I read the proposal is that we want to allow setting both, not
require it. If I misunderstood then please ignore my comments.

When I first heard about this proposal it seemed like a nice way for
frameworks and executors to start defining a more structured contract _if
they wanted to_, while still allowing others to keep passing their (from
Mesos's view) unstructured data as long as both sides agree on the
serialization / de-serialization.

Does this still seem like an issue if it's not a requirement but just a
broadening of possibilities?

—
*Joris Van Remoortere*
Mesosphere

On Wed, Nov 2, 2016 at 1:49 PM, Zameer Manji  wrote:

> Joris,
>
> You make a good point. However, I'm not convinced that `CommandInfo` should
> be the well defined construct that people use. Can you please describe
> different custom executors, and the overlap between them and how
> CommandInfo will reduce that overlap? I'm having a hard time seeing where
> CommandInfo will solve all of their cases.
>
> Consider the cause of Thermos (Aurora's Executor), it could never use a
> `CommandInfo` struct because it executes a processes graph instead of a
> single command.
>
> If the project wants to go down this path, I think generalizing
> `CommandInfo` that could capture more cases (ie multiple commands or a
> graph of commands) would be a better first step.
>
> What do you think?
>
> On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> jo...@mesosphere.io>
> wrote:
>
> > I do think it would be valuable to have a more well defined contract
> > between frameworks and custom executors.
> >
> > As Zameer pointed out a specific framework and accompanying custom
> executor
> > can decide to do that in the data bytes; however, if we started building
> > out a few different flavors of executors then it would be great for there
> > to be standard way to pass command information to them.
> >
> > The current model works well in a 1-1 mapping between framework and
> > executor binaries. In a world where that is 1-N it means all N executors
> > have to use the same method of passing the command.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji  wrote:
> >
> > > I'm not convinced this is a valid use case.
> > >
> > > Mesos is supposed to be a generic kernel for launching "tasks",
> whatever
> > > they might be.
> > >
> > > In some cases it is useful to launch an executable, in other cases it
> > might
> > > be useful to launch a series of executables, and in some other cases it
> > > might be useful to spawn a thread to do some work. Whatever that might
> > be,
> > > it doesn't matter to Mesos and the executor and framework are free to
> > > establish a contract in `ExecutorInfo.data`, completely independent of
> > the
> > > Mesos API.
> > >
> > > I think formalizing this contract between executors and frameworks via
> > > CommandInfo is going to introduce more problems than what they solve.
> If
> > > the CommandInfo struct is useful, frameworks and executors can just
> stuff
> > > that into ExecutorInfo.data, however it's not something that they need
> to
> > > adhere too.
> > >
> > > What's the underlying motivation for this?
> > >
> > >
> > >
> > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent  wrote:
> > >
> > > > For command task, if its `ExecutorInfo` would set with
> > `CommandExecutor`
> > > as
> > > > well?
> > > >
> > > > Some tickets may relate to this.
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > >
> > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are contemplating whether to allow both CommandInfo and
> > ExecutorInfo
> > > > on
> > > > > TaskInfo (MESOS-6294  > jira/browse/MESOS-6294
> > > > >).
> > > > > Currently we only allow one or the other. The motivation is to
> allow
> > > > custom
> > > > > executors a more structured way to pass information (e.g, command)
> > > about
> > > > > Task. Right now custom executors have to get this data via
> > > > `TaskInfo.bytes`
> > > > > which is not ideal.
> > > > >
> > > > > Are there any custom executors out there that crash if they get
> Tasks
> > > > with
> > > > > CommandInfo set?
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vinod
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > > > --
> > > > Zameer Manji
> > > >
> > >
> >
> > --
> > Zameer Manji
> >
>

Re: Test failures in Apache Jenkins

[Alex Rukletsov]

Yes, those were direct links. The source of *some* failures is probably VM
lags, which were reported in
https://issues.apache.org/jira/browse/INFRA-12852. I suggest we wait for a
resolution from Infra and see whether it helps and to what extent.

On Fri, Nov 4, 2016 at 12:42 AM, Benjamin Mahler  wrote:

> Hm.. these links are all broken, were you linking to jenkins logs directly?
> They get garbage collected rather quickly.
>
> On Mon, Oct 31, 2016 at 1:47 AM, Alex Rukletsov 
> wrote:
>
> > Folks,
> >
> > I observe a lot of flaky tests in Apache Jenkins. They seem rather random
> > and not tied to particular machines (saw failures on H1 and on H2).
> > Moreover, there are no tickets for them and I haven't seen any of those
> > failures in our internal CI.
> >
> > Does anyone have an idea about any recent changes in test harness,
> > libprocess or whatever that could lead to this? It's probably not related
> > to MESOS-6180 ,
> because
> > not all failures are future timeout induced.
> >
> > For example, in the last day I saw these guys failing:
> > ReconciliationTest.RecoveredAgent [1]
> > MasterTest.TaskLabels [2]
> > RoleTest.ImplicitRoleRegister [3]
> > ReconciliationTest.ImplicitTerminalTask [4]
> > ReservationTest.BadACLDropReserve [5]
> > ReservationTest.CompatibleCheckpointedResources [6]
> > ContentType/SchedulerHttpApiTest.Subscribe/0 [7]
> >
> > [1] https://goo.gl/cs88BD
> > [2] https://goo.gl/gTzKUV
> > [3] https://goo.gl/7pGaQG
> > [4] https://goo.gl/ccq38D
> > [5] https://goo.gl/0R1eOO
> > [6] https://goo.gl/xKQzUt
> > [7] https://goo.gl/HZmiGJ
> >
>

[VOTE] Release Apache Mesos 1.1.0 (rc3)

[Till Toenshoff]

Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.1.0.


1.1.0 includes the following:

  * [MESOS-2449] - **Experimental** support for launching a group of tasks
via a new `LAUNCH_GROUP` Offer operation. Mesos will guarantee that either
all tasks or none of the tasks in the group are delivered to the executor.
Executors receive the task group via a new `LAUNCH_GROUP` event.

  * [MESOS-2533] - **Experimental** support for HTTP and HTTPS health checks.
Executors may now use the updated `HealthCheck` protobuf to implement
HTTP(S) health checks. Both default executors (command and docker) leverage
`curl` binary for sending HTTP(S) requests and connect to `127.0.0.1`,
hence a task must listen on all interfaces. On Linux, for BRIDGE and USER
modes, docker executor enters the task's network namespace.

  * [MESOS-3421] - **Experimental** Support sharing of resources across
containers. Currently persistent volumes are the only resources allowed to
be shared.

  * [MESOS-3567] - **Experimental** support for TCP health checks. Executors
may now use the updated `HealthCheck` protobuf to implement TCP health
checks. Both default executors (command and docker) connect to `127.0.0.1`,
hence a task must listen on all interfaces. On Linux, for BRIDGE and USER
modes, docker executor enters the task's network namespace.

  * [MESOS-4324] - Allow tasks to access persistent volumes in either a
read-only or read-write manner. Using a volume in read-only mode can
simplify sharing that volume between multiple tasks on the same agent.

  * [MESOS-5275] - **Experimental** support for linux capabilities. Frameworks
or operators now have fine-grained control over the capabilities that a
container may have. This allows a container to run as root, but not have all
the privileges associated with the root user (e.g., CAP_SYS_ADMIN).

  * [MESOS-5344] - **Experimental** support for partition-aware Mesos
frameworks. In previous Mesos releases, when an agent is partitioned from
the master and then reregisters with the cluster, all tasks running on the
agent are terminated and the agent is shutdown. In Mesos 1.1, partitioned
agents will no longer be shutdown when they reregister with the master. By
default, tasks running on such agents will still be killed (for backward
compatibility); however, frameworks can opt-in to the new PARTITION_AWARE
capability. If they do this, their tasks will not be killed when a partition
is healed. This allows frameworks to define their own policies for how to
handle partitioned tasks. Enabling the PARTITION_AWARE capability also
introduces a new set of task states: TASK_UNREACHABLE, TASK_DROPPED,
TASK_GONE, TASK_GONE_BY_OPERATOR, and TASK_UNKNOWN. These new states are
intended to eventually replace the TASK_LOST state.

  * [MESOS-5788] - **Experimental** support for Java scheduler adapter. This
adapter allows framework developers to toggle between the old/new API
(driver/scheduler library) implementations, thereby allowing them to easily
transition their frameworks to the new v1 Scheduler API.

  * [MESOS-6014] - **Experimental** A new port-mapper CNI plugin, the
`mesos-cni-port-mapper` has been introduced. For Mesos containers, with the
CNI port-mapper plugin, users can now expose container ports through host
ports using DNAT. This is especially useful when Mesos containers are
attached to isolated CNI networks such as private bridge networks, and the
services running in the container needs to be exposed outside these
isolated networks.

  * [MESOS-6077] - **Experimental** A new default executor is introduced which
frameworks can use to launch task groups as nested containers. All the
nested containers share resources likes cpu, memory, network and volumes.

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.1.0-rc3


The candidate for Mesos 1.1.0 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.1.0-rc3/mesos-1.1.0.tar.gz

The tag to be voted on is 1.1.0-rc3:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.1.0-rc3

The MD5 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.1.0-rc3/mesos-1.1.0.tar.gz.md5

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.1.0-rc3/mesos-1.1.0.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is up in Maven in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1166

Please vote on releasing this package as Apache Mesos 1.1.