Re: Changing mesos slave configuration
We are adding better support for systemd in 0.25. The ticket is MESOS-3425. Naturally this is still somewhat experimental, but we would love your feedback. We will add some documentation on recommended setups on systemd. With the changes going into 0.25 you should be able to launch your slave with > Killmode=control-groups > Delegate=true The slave should be able to be restarted while the executors remain running un-interrupted. This requires systemd version 218+, or a patched implementation of prior version. You can see the related ticket for more information about supported patched packages by RHEL, etc.: MESOS-3352 Joris On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins wrote: > Are you using systemd? There is a known issue with slave recovery on > systemd. I'm on mobile or I would link you to the last thread around this > but there is a line you can add to the config that is supposed to fix it. > Whether it will fix it is another matter. I am fighting this issue at work > myself. > On Sep 23, 2015 7:53 AM, "Paul Bell" wrote: > >> Hi Pradeep, >> >> Perhaps I am speaking to a slightly different point, but when I change >> /etc/default/mesos-slave to add a new attribute, I have to remove file >> /tmp/mesos/meta/slaves/latest. >> >> IIRC, mesos-slave itself, in failing to start after such a change, tells >> me to do this: >> >> rm -f /tmp/mesos/meta/slaves/latest >> >> >> But I know of no way to make such configuration changes without downtime. >> And I'd very much like it if Mesos supported such dynamic changes. I >> suppose this would require that the agent consult its default file on >> demand, rather than once at start-up. >> >> Cordially, >> >> Paul >> >> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < >> pradeep.chhetr...@gmail.com> wrote: >> >>> Hello all, >>> >>> I have often faced this problem that whenever i try to add some >>> configuration parameter to mesos-slave or change any configuration (eg. add >>> a new attribute in mesos-slave), the mesos slave doesnt come up on restart. >>> I have to delete the slave.info file and then restart the slave but it >>> ends up killing all the docker containers started using mesos. >>> >>> I was trying to figure out the best way to make such changes without >>> making any downtime. >>> >>> Thank you. >>> >>> -- >>> Pradeep Chhetri >>> >> >>
Re: Changing mesos slave configuration
Thank you for the issue link. I will go through to understand which configuration changes can be done with and without recovery. On Wed, Sep 23, 2015 at 4:25 PM, Vinod Kone wrote: > It's not yet possible to make certain slave configuration changes while > making recovery (reconnecting with old executors) work. > > See https://issues.apache.org/jira/browse/MESOS-1739 and attached tickets > for details. > > On Wed, Sep 23, 2015 at 7:37 AM, Pradeep Chhetri < > pradeep.chhetr...@gmail.com> wrote: > >> Thank you for the replies. >> >> Paul, I am talking the about the same directory. There is a file named >> slave.info inside /tmp/mesos/meta/slaves/latest and this needs to be >> cleaned before starting mesos slave with a configuration change. >> >> No i am not using systemd. It is basically sysvinit which is spawning the >> mesos-slave. >> >> Thank you. >> >> On Wed, Sep 23, 2015 at 1:35 PM, craig w wrote: >> >>> I believe Brian might be referring to the "KillMode" in the systemd unit >>> file: >>> >>> # the default is cgroup, which means kill all processes >>> # in the control group of this process, which is not >>> # what you'd want >>> KillMode=process >>> >>> On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins >>> wrote: >>> Are you using systemd? There is a known issue with slave recovery on systemd. I'm on mobile or I would link you to the last thread around this but there is a line you can add to the config that is supposed to fix it. Whether it will fix it is another matter. I am fighting this issue at work myself. On Sep 23, 2015 7:53 AM, "Paul Bell" wrote: > Hi Pradeep, > > Perhaps I am speaking to a slightly different point, but when I change > /etc/default/mesos-slave to add a new attribute, I have to remove file > /tmp/mesos/meta/slaves/latest. > > IIRC, mesos-slave itself, in failing to start after such a change, > tells me to do this: > > rm -f /tmp/mesos/meta/slaves/latest > > > But I know of no way to make such configuration changes without > downtime. And I'd very much like it if Mesos supported such dynamic > changes. I suppose this would require that the agent consult its default > file on demand, rather than once at start-up. > > Cordially, > > Paul > > On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < > pradeep.chhetr...@gmail.com> wrote: > >> Hello all, >> >> I have often faced this problem that whenever i try to add some >> configuration parameter to mesos-slave or change any configuration (eg. >> add >> a new attribute in mesos-slave), the mesos slave doesnt come up on >> restart. >> I have to delete the slave.info file and then restart the slave but >> it ends up killing all the docker containers started using mesos. >> >> I was trying to figure out the best way to make such changes without >> making any downtime. >> >> Thank you. >> >> -- >> Pradeep Chhetri >> > > >>> >>> >>> -- >>> >>> https://github.com/mindscratch >>> https://www.google.com/+CraigWickesser >>> https://twitter.com/mind_scratch >>> https://twitter.com/craig_links >>> >>> >> >> >> -- >> Pradeep Chhetri >> >> In the world of Linux, who needs Windows and Gates... >> > > -- Pradeep Chhetri In the world of Linux, who needs Windows and Gates...
Re: Changing mesos slave configuration
It's not yet possible to make certain slave configuration changes while making recovery (reconnecting with old executors) work. See https://issues.apache.org/jira/browse/MESOS-1739 and attached tickets for details. On Wed, Sep 23, 2015 at 7:37 AM, Pradeep Chhetri < pradeep.chhetr...@gmail.com> wrote: > Thank you for the replies. > > Paul, I am talking the about the same directory. There is a file named > slave.info inside /tmp/mesos/meta/slaves/latest and this needs to be > cleaned before starting mesos slave with a configuration change. > > No i am not using systemd. It is basically sysvinit which is spawning the > mesos-slave. > > Thank you. > > On Wed, Sep 23, 2015 at 1:35 PM, craig w wrote: > >> I believe Brian might be referring to the "KillMode" in the systemd unit >> file: >> >> # the default is cgroup, which means kill all processes >> # in the control group of this process, which is not >> # what you'd want >> KillMode=process >> >> On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins wrote: >> >>> Are you using systemd? There is a known issue with slave recovery on >>> systemd. I'm on mobile or I would link you to the last thread around this >>> but there is a line you can add to the config that is supposed to fix it. >>> Whether it will fix it is another matter. I am fighting this issue at work >>> myself. >>> On Sep 23, 2015 7:53 AM, "Paul Bell" wrote: >>> Hi Pradeep, Perhaps I am speaking to a slightly different point, but when I change /etc/default/mesos-slave to add a new attribute, I have to remove file /tmp/mesos/meta/slaves/latest. IIRC, mesos-slave itself, in failing to start after such a change, tells me to do this: rm -f /tmp/mesos/meta/slaves/latest But I know of no way to make such configuration changes without downtime. And I'd very much like it if Mesos supported such dynamic changes. I suppose this would require that the agent consult its default file on demand, rather than once at start-up. Cordially, Paul On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < pradeep.chhetr...@gmail.com> wrote: > Hello all, > > I have often faced this problem that whenever i try to add some > configuration parameter to mesos-slave or change any configuration (eg. > add > a new attribute in mesos-slave), the mesos slave doesnt come up on > restart. > I have to delete the slave.info file and then restart the slave but > it ends up killing all the docker containers started using mesos. > > I was trying to figure out the best way to make such changes without > making any downtime. > > Thank you. > > -- > Pradeep Chhetri > >> >> >> -- >> >> https://github.com/mindscratch >> https://www.google.com/+CraigWickesser >> https://twitter.com/mind_scratch >> https://twitter.com/craig_links >> >> > > > -- > Pradeep Chhetri > > In the world of Linux, who needs Windows and Gates... >
Re: Changing mesos slave configuration
Thank you for the replies. Paul, I am talking the about the same directory. There is a file named slave.info inside /tmp/mesos/meta/slaves/latest and this needs to be cleaned before starting mesos slave with a configuration change. No i am not using systemd. It is basically sysvinit which is spawning the mesos-slave. Thank you. On Wed, Sep 23, 2015 at 1:35 PM, craig w wrote: > I believe Brian might be referring to the "KillMode" in the systemd unit > file: > > # the default is cgroup, which means kill all processes > # in the control group of this process, which is not > # what you'd want > KillMode=process > > On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins wrote: > >> Are you using systemd? There is a known issue with slave recovery on >> systemd. I'm on mobile or I would link you to the last thread around this >> but there is a line you can add to the config that is supposed to fix it. >> Whether it will fix it is another matter. I am fighting this issue at work >> myself. >> On Sep 23, 2015 7:53 AM, "Paul Bell" wrote: >> >>> Hi Pradeep, >>> >>> Perhaps I am speaking to a slightly different point, but when I change >>> /etc/default/mesos-slave to add a new attribute, I have to remove file >>> /tmp/mesos/meta/slaves/latest. >>> >>> IIRC, mesos-slave itself, in failing to start after such a change, tells >>> me to do this: >>> >>> rm -f /tmp/mesos/meta/slaves/latest >>> >>> >>> But I know of no way to make such configuration changes without >>> downtime. And I'd very much like it if Mesos supported such dynamic >>> changes. I suppose this would require that the agent consult its default >>> file on demand, rather than once at start-up. >>> >>> Cordially, >>> >>> Paul >>> >>> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < >>> pradeep.chhetr...@gmail.com> wrote: >>> Hello all, I have often faced this problem that whenever i try to add some configuration parameter to mesos-slave or change any configuration (eg. add a new attribute in mesos-slave), the mesos slave doesnt come up on restart. I have to delete the slave.info file and then restart the slave but it ends up killing all the docker containers started using mesos. I was trying to figure out the best way to make such changes without making any downtime. Thank you. -- Pradeep Chhetri >>> >>> > > > -- > > https://github.com/mindscratch > https://www.google.com/+CraigWickesser > https://twitter.com/mind_scratch > https://twitter.com/craig_links > > -- Pradeep Chhetri In the world of Linux, who needs Windows and Gates...
Re: Changing mesos slave configuration
I believe Brian might be referring to the "KillMode" in the systemd unit file: # the default is cgroup, which means kill all processes # in the control group of this process, which is not # what you'd want KillMode=process On Wed, Sep 23, 2015 at 8:11 AM, Brian Devins wrote: > Are you using systemd? There is a known issue with slave recovery on > systemd. I'm on mobile or I would link you to the last thread around this > but there is a line you can add to the config that is supposed to fix it. > Whether it will fix it is another matter. I am fighting this issue at work > myself. > On Sep 23, 2015 7:53 AM, "Paul Bell" wrote: > >> Hi Pradeep, >> >> Perhaps I am speaking to a slightly different point, but when I change >> /etc/default/mesos-slave to add a new attribute, I have to remove file >> /tmp/mesos/meta/slaves/latest. >> >> IIRC, mesos-slave itself, in failing to start after such a change, tells >> me to do this: >> >> rm -f /tmp/mesos/meta/slaves/latest >> >> >> But I know of no way to make such configuration changes without downtime. >> And I'd very much like it if Mesos supported such dynamic changes. I >> suppose this would require that the agent consult its default file on >> demand, rather than once at start-up. >> >> Cordially, >> >> Paul >> >> On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < >> pradeep.chhetr...@gmail.com> wrote: >> >>> Hello all, >>> >>> I have often faced this problem that whenever i try to add some >>> configuration parameter to mesos-slave or change any configuration (eg. add >>> a new attribute in mesos-slave), the mesos slave doesnt come up on restart. >>> I have to delete the slave.info file and then restart the slave but it >>> ends up killing all the docker containers started using mesos. >>> >>> I was trying to figure out the best way to make such changes without >>> making any downtime. >>> >>> Thank you. >>> >>> -- >>> Pradeep Chhetri >>> >> >> -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: Changing mesos slave configuration
Are you using systemd? There is a known issue with slave recovery on systemd. I'm on mobile or I would link you to the last thread around this but there is a line you can add to the config that is supposed to fix it. Whether it will fix it is another matter. I am fighting this issue at work myself. On Sep 23, 2015 7:53 AM, "Paul Bell" wrote: > Hi Pradeep, > > Perhaps I am speaking to a slightly different point, but when I change > /etc/default/mesos-slave to add a new attribute, I have to remove file > /tmp/mesos/meta/slaves/latest. > > IIRC, mesos-slave itself, in failing to start after such a change, tells > me to do this: > > rm -f /tmp/mesos/meta/slaves/latest > > > But I know of no way to make such configuration changes without downtime. > And I'd very much like it if Mesos supported such dynamic changes. I > suppose this would require that the agent consult its default file on > demand, rather than once at start-up. > > Cordially, > > Paul > > On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < > pradeep.chhetr...@gmail.com> wrote: > >> Hello all, >> >> I have often faced this problem that whenever i try to add some >> configuration parameter to mesos-slave or change any configuration (eg. add >> a new attribute in mesos-slave), the mesos slave doesnt come up on restart. >> I have to delete the slave.info file and then restart the slave but it >> ends up killing all the docker containers started using mesos. >> >> I was trying to figure out the best way to make such changes without >> making any downtime. >> >> Thank you. >> >> -- >> Pradeep Chhetri >> > >
Re: Changing mesos slave configuration
Hi Pradeep, Perhaps I am speaking to a slightly different point, but when I change /etc/default/mesos-slave to add a new attribute, I have to remove file /tmp/mesos/meta/slaves/latest. IIRC, mesos-slave itself, in failing to start after such a change, tells me to do this: rm -f /tmp/mesos/meta/slaves/latest But I know of no way to make such configuration changes without downtime. And I'd very much like it if Mesos supported such dynamic changes. I suppose this would require that the agent consult its default file on demand, rather than once at start-up. Cordially, Paul On Wed, Sep 23, 2015 at 4:41 AM, Pradeep Chhetri < pradeep.chhetr...@gmail.com> wrote: > Hello all, > > I have often faced this problem that whenever i try to add some > configuration parameter to mesos-slave or change any configuration (eg. add > a new attribute in mesos-slave), the mesos slave doesnt come up on restart. > I have to delete the slave.info file and then restart the slave but it > ends up killing all the docker containers started using mesos. > > I was trying to figure out the best way to make such changes without > making any downtime. > > Thank you. > > -- > Pradeep Chhetri >
Changing mesos slave configuration
Hello all, I have often faced this problem that whenever i try to add some configuration parameter to mesos-slave or change any configuration (eg. add a new attribute in mesos-slave), the mesos slave doesnt come up on restart. I have to delete the slave.info file and then restart the slave but it ends up killing all the docker containers started using mesos. I was trying to figure out the best way to make such changes without making any downtime. Thank you. -- Pradeep Chhetri