Re: Changing mesos slave configuration

[Brian Devins]

Are you using systemd? There is a known issue with slave recovery on
systemd. I'm on mobile or I would link you to the last thread around this
but there is a line you can add to the config that is supposed to fix it.
Whether it will fix it is another matter. I am fighting this issue at work
myself.
On Sep 23, 2015 7:53 AM, "Paul Bell"  wrote:

> Hi Pradeep,
>
> Perhaps I am speaking to a slightly different point, but when I change
> /etc/default/mesos-slave to add a new attribute, I have to remove file
> /tmp/mesos/meta/slaves/latest.
>
> IIRC, mesos-slave itself, in failing to start after such a change, tells
> me to do this:
>
> rm -f /tmp/mesos/meta/slaves/latest
>
>
> But I know of no way to make such configuration changes without downtime.
> And I'd very much like it if Mesos supported such dynamic changes. I
> suppose this would require that the agent consult its default file on
> demand, rather than once at start-up.
>
> Cordially,
>
> Paul
>
> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri <
> pradeep.chhetr...@gmail.com> wrote:
>
>> Hello all,
>>
>> I have often faced this problem that whenever i try to add some
>> configuration parameter to mesos-slave or change any configuration (eg. add
>> a new attribute in mesos-slave), the mesos slave doesnt come up on restart.
>> I have to delete the slave.info file and then restart the slave but it
>> ends up killing all the docker containers started using mesos.
>>
>> I was trying to figure out the best way to make such changes without
>> making any downtime.
>>
>> Thank you.
>>
>> --
>> Pradeep Chhetri
>>
>
>

Re: Changing mesos slave configuration

[Paul Bell]

Hi Pradeep,

Perhaps I am speaking to a slightly different point, but when I change
/etc/default/mesos-slave to add a new attribute, I have to remove file
/tmp/mesos/meta/slaves/latest.

IIRC, mesos-slave itself, in failing to start after such a change, tells me
to do this:

rm -f /tmp/mesos/meta/slaves/latest


But I know of no way to make such configuration changes without downtime.
And I'd very much like it if Mesos supported such dynamic changes. I
suppose this would require that the agent consult its default file on
demand, rather than once at start-up.

Cordially,

Paul

On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri <
pradeep.chhetr...@gmail.com> wrote:

> Hello all,
>
> I have often faced this problem that whenever i try to add some
> configuration parameter to mesos-slave or change any configuration (eg. add
> a new attribute in mesos-slave), the mesos slave doesnt come up on restart.
> I have to delete the slave.info file and then restart the slave but it
> ends up killing all the docker containers started using mesos.
>
> I was trying to figure out the best way to make such changes without
> making any downtime.
>
> Thank you.
>
> --
> Pradeep Chhetri
>

Re: Changing mesos slave configuration

[craig w]

I believe Brian might be referring to the "KillMode" in the systemd unit
file:

# the default is cgroup, which means kill all processes
# in the control group of this process, which is not
# what you'd want
KillMode=process

On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins  wrote:

> Are you using systemd? There is a known issue with slave recovery on
> systemd. I'm on mobile or I would link you to the last thread around this
> but there is a line you can add to the config that is supposed to fix it.
> Whether it will fix it is another matter. I am fighting this issue at work
> myself.
> On Sep 23, 2015 7:53 AM, "Paul Bell"  wrote:
>
>> Hi Pradeep,
>>
>> Perhaps I am speaking to a slightly different point, but when I change
>> /etc/default/mesos-slave to add a new attribute, I have to remove file
>> /tmp/mesos/meta/slaves/latest.
>>
>> IIRC, mesos-slave itself, in failing to start after such a change, tells
>> me to do this:
>>
>> rm -f /tmp/mesos/meta/slaves/latest
>>
>>
>> But I know of no way to make such configuration changes without downtime.
>> And I'd very much like it if Mesos supported such dynamic changes. I
>> suppose this would require that the agent consult its default file on
>> demand, rather than once at start-up.
>>
>> Cordially,
>>
>> Paul
>>
>> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri <
>> pradeep.chhetr...@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> I have often faced this problem that whenever i try to add some
>>> configuration parameter to mesos-slave or change any configuration (eg. add
>>> a new attribute in mesos-slave), the mesos slave doesnt come up on restart.
>>> I have to delete the slave.info file and then restart the slave but it
>>> ends up killing all the docker containers started using mesos.
>>>
>>> I was trying to figure out the best way to make such changes without
>>> making any downtime.
>>>
>>> Thank you.
>>>
>>> --
>>> Pradeep Chhetri
>>>
>>
>>


-- 

https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links

Re: Changing mesos slave configuration

[Pradeep Chhetri]

Thank you for the replies.

Paul, I am talking the about the same directory. There is a file named
slave.info inside /tmp/mesos/meta/slaves/latest and this needs to be
cleaned before starting mesos slave with a configuration change.

No i am not using systemd. It is basically sysvinit which is spawning the
mesos-slave.

Thank you.

On Wed, Sep 23, 2015 at 1:35 PM, craig w  wrote:

> I believe Brian might be referring to the "KillMode" in the systemd unit
> file:
>
> # the default is cgroup, which means kill all processes
> # in the control group of this process, which is not
> # what you'd want
> KillMode=process
>
> On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins  wrote:
>
>> Are you using systemd? There is a known issue with slave recovery on
>> systemd. I'm on mobile or I would link you to the last thread around this
>> but there is a line you can add to the config that is supposed to fix it.
>> Whether it will fix it is another matter. I am fighting this issue at work
>> myself.
>> On Sep 23, 2015 7:53 AM, "Paul Bell"  wrote:
>>
>>> Hi Pradeep,
>>>
>>> Perhaps I am speaking to a slightly different point, but when I change
>>> /etc/default/mesos-slave to add a new attribute, I have to remove file
>>> /tmp/mesos/meta/slaves/latest.
>>>
>>> IIRC, mesos-slave itself, in failing to start after such a change, tells
>>> me to do this:
>>>
>>> rm -f /tmp/mesos/meta/slaves/latest
>>>
>>>
>>> But I know of no way to make such configuration changes without
>>> downtime. And I'd very much like it if Mesos supported such dynamic
>>> changes. I suppose this would require that the agent consult its default
>>> file on demand, rather than once at start-up.
>>>
>>> Cordially,
>>>
>>> Paul
>>>
>>> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri <
>>> pradeep.chhetr...@gmail.com> wrote:
>>>
 Hello all,

 I have often faced this problem that whenever i try to add some
 configuration parameter to mesos-slave or change any configuration (eg. add
 a new attribute in mesos-slave), the mesos slave doesnt come up on restart.
 I have to delete the slave.info file and then restart the slave but it
 ends up killing all the docker containers started using mesos.

 I was trying to figure out the best way to make such changes without
 making any downtime.

 Thank you.

 --
 Pradeep Chhetri

>>>
>>>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
>


-- 
Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

Re: Changing mesos slave configuration

[Joris Van Remoortere]

We are adding better support for systemd in 0.25. The ticket is MESOS-3425.
Naturally this is still somewhat experimental, but we would love your
feedback.
We will add some documentation on recommended setups on systemd.

With the changes going into 0.25 you should be able to launch your slave
with

> Killmode=control-groups
> Delegate=true


The slave should be able to be restarted while the executors remain running
un-interrupted.

This requires systemd version 218+, or a patched implementation of prior
version. You can see the related ticket for more information about
supported patched packages by RHEL, etc.: MESOS-3352

Joris

On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins  wrote:

> Are you using systemd? There is a known issue with slave recovery on
> systemd. I'm on mobile or I would link you to the last thread around this
> but there is a line you can add to the config that is supposed to fix it.
> Whether it will fix it is another matter. I am fighting this issue at work
> myself.
> On Sep 23, 2015 7:53 AM, "Paul Bell"  wrote:
>
>> Hi Pradeep,
>>
>> Perhaps I am speaking to a slightly different point, but when I change
>> /etc/default/mesos-slave to add a new attribute, I have to remove file
>> /tmp/mesos/meta/slaves/latest.
>>
>> IIRC, mesos-slave itself, in failing to start after such a change, tells
>> me to do this:
>>
>> rm -f /tmp/mesos/meta/slaves/latest
>>
>>
>> But I know of no way to make such configuration changes without downtime.
>> And I'd very much like it if Mesos supported such dynamic changes. I
>> suppose this would require that the agent consult its default file on
>> demand, rather than once at start-up.
>>
>> Cordially,
>>
>> Paul
>>
>> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri <
>> pradeep.chhetr...@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> I have often faced this problem that whenever i try to add some
>>> configuration parameter to mesos-slave or change any configuration (eg. add
>>> a new attribute in mesos-slave), the mesos slave doesnt come up on restart.
>>> I have to delete the slave.info file and then restart the slave but it
>>> ends up killing all the docker containers started using mesos.
>>>
>>> I was trying to figure out the best way to make such changes without
>>> making any downtime.
>>>
>>> Thank you.
>>>
>>> --
>>> Pradeep Chhetri
>>>
>>
>>