Re: Docker Executor in Mesos

[Du, Fan]

So why not use one executor to launch docker tasks?


Each task resides(or runs to be more precisely) in its own docker container,
Every docker container is an executor by its nature.
If you launch 6 instances of task, there will be 6 docker 
containers(docker ps).


I'm not sure what's the intention of "use one executor to launch docker 
tasks",

and how to do this.

On 2015/12/8 14:05, Klaus Ma wrote:

Hi team,

Currently, if we run docker in mesos, we'll start docker-executor, "docker
run" and container in slave hosts. So why not use one executor to launch
docker tasks? One reason I can image is compatibility of docker API. If
there're thousands of tasks in a powerful task, do you get docker startup
performance issue?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Docker Executor in Mesos

[Klaus Ma]

Hi team,

Currently, if we run docker in mesos, we'll start docker-executor, "docker
run" and container in slave hosts. So why not use one executor to launch
docker tasks? One reason I can image is compatibility of docker API. If
there're thousands of tasks in a powerful task, do you get docker startup
performance issue?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

[VOTE] Release Apache Mesos 0.26.0 (rc4)

[Till Toenshoff]

Hi friends,

we had noticed some discrepancies between the V0 API and the V1 API,
hence we had to create a new release candidate even after the voting of
0.26.0-rc3 had officially ended. Sorry for that!

Please vote on releasing the following candidate as Apache Mesos 0.26.0.

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.26.0-rc4


The candidate for Mesos 0.26.0 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc4/mesos-0.26.0.tar.gz

The tag to be voted on is 0.26.0-rc4:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.26.0-rc4

The MD5 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc4/mesos-0.26.0.tar.gz.md5

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc4/mesos-0.26.0.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is up in Maven in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1093

Please vote on releasing this package as Apache Mesos 0.26.0!

The vote is open until Fri Dec 11 04:50:51 CET 2015 and passes if a majority of 
at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 0.26.0
[ ] -1 Do not release this package because ...

Thanks,
Bernd & Till

Re: [VOTE] Release Apache Mesos 0.26.0 (rc3)

[Bernd Mathiske]

We found an issue with the v1 API, which is not completely on par with the 
legacy mesos.proto. Spinning RC4 shortly…

> On Dec 7, 2015, at 10:48 AM, Bernd Mathiske  wrote:
> 
> +1 (binding)
> 
> Ubuntu 14 (clean without SSL, a few known flaky tests with SSL, all analyzed 
> and deemed non-blockers)
> CentOS 6.6 (a few known flaky tests with SSL, all analyzed and deemed 
> non-blockers)
> 
>> On Dec 4, 2015, at 4:52 AM, Benjamin Mahler  
>> wrote:
>> 
>> +1 (binding) tests pass on OS X 10.11.1 with both SSL and non-SSL
>> configurations.
>> 
>> Some feedback from framework developers would be great here.
>> 
>> Agreed that MESOS-3973  is
>> not a blocker, given it also occurs on 0.26.0 all the way back to 0.21.0.
>> 
>> 
>> On Wed, Dec 2, 2015 at 9:01 AM, Bernd Mathiske  wrote:
>> 
>>> We are still working on that, but we do not regard "make distcheck" on Mac
>>> as blocker. Other opinions?
>>> 
>>> On Dec 2, 2015, at 2:27 PM, Alex Rukletsov  wrote:
>>> 
>>> `make check -j7` — OK
>>> `make distcheck -j7` — fails, probably MESOS-3973
>>> , see hints below.
>>> 
>>> Both on Mac OS 10.10.4
>>> 
>>> I see the following lines in the log:
>>> ...
>>> libtool: warning: 'libmesos.la' has not been installed in
>>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>>> libtool: warning: 'libmesos.la' has not been installed in
>>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>>> ...
>>> libtool: warning: 'libmesos.la' has not been installed in
>>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>>> libtool: warning: 'libmesos.la' has not been installed in
>>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>>> ...
>>> Cannot uninstall requirement mesos, not installed
>>> Cannot uninstall requirement mesos.cli, not installed
>>> Cannot uninstall requirement mesos.interface, not installed
>>> Cannot uninstall requirement mesos.native, not installed
>>> ERROR: files left after uninstall:
>>> ...
>>> 
>>> On Tue, Dec 1, 2015 at 8:49 PM, Till Toenshoff  wrote:
>>> 
 Hi friends,
 
 Please vote on releasing the following candidate as Apache Mesos 0.26.0.
 
 The CHANGELOG for the release is available at:
 
 https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.26.0-rc3
 
 
 
 The candidate for Mesos 0.26.0 release is available at:
 
 https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc3/mesos-0.26.0.tar.gz
 
 The tag to be voted on is 0.26.0-rc3:
 https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.26.0-rc3
 
 The MD5 checksum of the tarball can be found at:
 
 https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc3/mesos-0.26.0.tar.gz.md5
 
 The signature of the tarball can be found at:
 
 https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc3/mesos-0.26.0.tar.gz.asc
 
 The PGP key used to sign the release is here:
 https://dist.apache.org/repos/dist/release/mesos/KEYS
 
 The JAR is up in Maven in a staging repository here:
 https://repository.apache.org/content/repositories/orgapachemesos-1091
 
 Please vote on releasing this package as Apache Mesos 0.26.0!
 
 The vote is open until Fri Dec  4 19:00:35 CET 2015 and passes if a
 majority of at least 3 +1 PMC votes are cast.
 
 [ ] +1 Release this package as Apache Mesos 0.26.0
 [ ] -1 Do not release this package because …
 
 Thanks,
 Bernd & Till
 
 
>>> 
>>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Please review design doc for task resizing

[Qian Zhang]

Thanks Niklas for your comments :-)

For your first comment, so you prefer option 2 in the design doc (i.e., add
resize as a new offer operation), right? Actually after more thinking, I
think if we want to support the following 2 use cases (especially the
second one) and OK to resize a task with an empty offer list (meaning we
may need to remove the check:
https://github.com/apache/mesos/blob/0.25.0/src/master/master.cpp#L2816 for
reducing resource from a task), then I also agree option 2 is the best.
1. Framework adds / reduces multiple resources for its task at the same
time.
2. Framework reserves the resources and resizes a task to use those
resources at the same time.

For your second comment, can you please clarify more about "should be
dictated by the resource type itself"? I see your comment in the design doc
"The resource type should dictate the sign of R_2 -R_1.", but I am not sure
what you meant about "R_2 -R_1". Actually there are still some discussion
about whether frameworks need to send desired resource (e.g., resize my
task to 8GB) or send the resource delta (e.g., add 2GB to my task) to
master, personally I prefer the later because the former can cause some
race condition that we can not handle, you can refer my comments in the
design doc for details.

And I have 2 more questions that I want to discuss with you:
1. David G raised a user story about framework should be able to resize its
executor, I think this should be a valid use case, but I would suggest us
to focus on task resizing in MVP and handle executor resizing in the
post-MVP, how do you think?
2. Do you think we need to involve executor in task resizing? E.g., let
slave send a message (e.g., RunTaskMessage) to executor so that executor
can do the actual resizing? The reason I raise this question is that I
think in some cases, executor needs to be aware of the resized resources,
e.g., framework adds a new port to a task, I think executor & task should
know such new port so that the task can start to use it. And in the
Kubernetes on Mesos case, user may want to resize a pod which is actually
created an managed by k8sm-executor, so it should be involved to resize the
resources of the pod.

Currently I do not have PoC implementation for my proposal yet, do you
recommend that we should have it now? Or after the design is close to be
finalized or at least after we make the decision among those 3 options
about scheduler API changes in the design doc?

I'd like to have an online sync up with you, can you please let me know
when you will be online in IRC usually? Or you prefer other ways to sync
up? I will try to catch you :-)


Thanks,
Qian


2015-12-05 7:03 GMT+08:00 Niklas Nielsen :

> Hi Qian,
>
> Thanks for the update and I apologize the response time.
>
> Do you have a PoC implementation of your proposal?
>
> I have trouble understanding the motivation of _not_ adding resizing as a
> usual operation. It seems much cleaner in my mind. To David G's and Alex
> R's comment: if you want to resize without an offer (during task
> shrinking), you could do it with an empty offer list. Giving up combining
> task resizing with the other operations (which will most likely scale with
> upcoming features) is a big loss, but maybe I am missing something.
>
> Secondly, whether the new desired resource shape requires growing and
> shrinking, I think should be dictated by the resource type itself rather
> than explicitly set by the framework writer. You have to do that math
> anyway to figure out whether the framework's request is valid, no?
>
> We can do a online sync soon, if you want to give a pitch on the design.
>
> Cheers,
> Niklas
>
>
> On Thu, Nov 19, 2015 at 6:34 AM, Qian Zhang  wrote:
>
> > Hi All,
> >
> > I am currently working on task resizing (MESOS-938), and have drafted a
> > design doc (see the link below).
> >
> >
> https://docs.google.com/document/d/15rVmS2AXLzTDSEugAVDxWuHFUentp82KhL2yzxBCsi8/edit?usp=sharing
> >
> >
> > Please feel free to review it, any comments are welcome, thanks!
> >
> >
> > Regards,
> > Qian
> >
>
>
>
> --
> Niklas
>

Re: [VOTE] Release Apache Mesos 0.26.0 (rc3)

[Bernd Mathiske]

+1 (binding)

Ubuntu 14 (clean without SSL, a few known flaky tests with SSL, all analyzed 
and deemed non-blockers)
CentOS 6.6 (a few known flaky tests with SSL, all analyzed and deemed 
non-blockers)

> On Dec 4, 2015, at 4:52 AM, Benjamin Mahler  wrote:
> 
> +1 (binding) tests pass on OS X 10.11.1 with both SSL and non-SSL
> configurations.
> 
> Some feedback from framework developers would be great here.
> 
> Agreed that MESOS-3973  is
> not a blocker, given it also occurs on 0.26.0 all the way back to 0.21.0.
> 
> 
> On Wed, Dec 2, 2015 at 9:01 AM, Bernd Mathiske  wrote:
> 
>> We are still working on that, but we do not regard "make distcheck" on Mac
>> as blocker. Other opinions?
>> 
>> On Dec 2, 2015, at 2:27 PM, Alex Rukletsov  wrote:
>> 
>> `make check -j7` — OK
>> `make distcheck -j7` — fails, probably MESOS-3973
>> , see hints below.
>> 
>> Both on Mac OS 10.10.4
>> 
>> I see the following lines in the log:
>> ...
>> libtool: warning: 'libmesos.la' has not been installed in
>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>> libtool: warning: 'libmesos.la' has not been installed in
>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>> ...
>> libtool: warning: 'libmesos.la' has not been installed in
>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>> libtool: warning: 'libmesos.la' has not been installed in
>> '/Users/alex/Projects/mesos/build/default/mesos-0.26.0/_inst/lib'
>> ...
>> Cannot uninstall requirement mesos, not installed
>> Cannot uninstall requirement mesos.cli, not installed
>> Cannot uninstall requirement mesos.interface, not installed
>> Cannot uninstall requirement mesos.native, not installed
>> ERROR: files left after uninstall:
>> ...
>> 
>> On Tue, Dec 1, 2015 at 8:49 PM, Till Toenshoff  wrote:
>> 
>>> Hi friends,
>>> 
>>> Please vote on releasing the following candidate as Apache Mesos 0.26.0.
>>> 
>>> The CHANGELOG for the release is available at:
>>> 
>>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.26.0-rc3
>>> 
>>> 
>>> 
>>> The candidate for Mesos 0.26.0 release is available at:
>>> 
>>> https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc3/mesos-0.26.0.tar.gz
>>> 
>>> The tag to be voted on is 0.26.0-rc3:
>>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.26.0-rc3
>>> 
>>> The MD5 checksum of the tarball can be found at:
>>> 
>>> https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc3/mesos-0.26.0.tar.gz.md5
>>> 
>>> The signature of the tarball can be found at:
>>> 
>>> https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc3/mesos-0.26.0.tar.gz.asc
>>> 
>>> The PGP key used to sign the release is here:
>>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>> 
>>> The JAR is up in Maven in a staging repository here:
>>> https://repository.apache.org/content/repositories/orgapachemesos-1091
>>> 
>>> Please vote on releasing this package as Apache Mesos 0.26.0!
>>> 
>>> The vote is open until Fri Dec  4 19:00:35 CET 2015 and passes if a
>>> majority of at least 3 +1 PMC votes are cast.
>>> 
>>> [ ] +1 Release this package as Apache Mesos 0.26.0
>>> [ ] -1 Do not release this package because …
>>> 
>>> Thanks,
>>> Bernd & Till
>>> 
>>> 
>> 
>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail