Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
On 25 Feb 2015, at 1:37 pm, Serge Dubrouski serge...@gmail.com wrote: Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea. Want to help make it work? :-) Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine? Probably because someone realised that they wanted to container-ize the software for creating containers and nesting them was too horrible to contemplate. On Mon, Feb 23, 2015 at 5:20 PM, Andrew Beekhof and...@beekhof.net wrote: On 10 Feb 2015, at 1:45 pm, Serge Dubrouski serge...@gmail.com wrote: Hello Steve, Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers. And in a few years they'll work out that they need some HA features and try to retrofit them :-) In the meantime pacemaker is actually rather good at managing containers already and knows a thing or two about HA and how to bring up a complex stack of services. The one thing that would be really interesting in this area is using the our policy engine as the kubernates scheduler. So many ideas and so little time. On Sat Feb 07 2015 at 1:19:15 PM Steven Dake (stdake) std...@cisco.com wrote: Hi, I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Serge Dubrouski. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
On 26 Feb 2015, at 8:51 am, Digimer li...@alteeve.ca wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 25/02/15 04:45 PM, David Vossel wrote: - Original Message - Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea. Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine? because soon everything will run in containers. Take a look at rhel atomic and the stuff coreos is doing. The only way pacemaker will exist on those distributions is if it lives in a super privileged container. If I can respectfully disagree, or at least provide a different opinion; I'm old enough to have seen many supposed game changers come and go. Maybe containers will be different, but to simply assume so at this stage and to base the future development of HA software on that assumption seems ... risky. I think there is an argument for supporting containers, sure, but not to base plans on the assumption that they will become the be-all and end-all. Key phrase: on these distros. David's not suggesting everyone run Pacemaker like this, only people that have bought into distros that drank the container brand cool-aid (probably out of a container!). - -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJU7kPSAAoJECChztQA3mh0lR4P/3JicT5jSk976BHqNcTPmWsM DHpumNORtpVPMaktC6lT6PRYeyFtYU2lqYgtBW9tpFbNai5h3qwhscV2O0mUON1L T4NGB1ZwlVAaf+ziZwxThGor/27hS29mHHVgaqR8ePhmjwxD4VX+A3f1S9QKu59B 0FuFLnWjYFMZwDV0CXGD5IxXf9aoPI47M/AwXE18Aj9TvuwblKw5z03VXUSp1kLh jFb8WcLjX+i11vXUnVnNsOHZfvAg8ejaA5NZLj4se0cMHwA0O3geAjGNAwqrACcp R0LRkxD0RBv1II3cx0Z4ZIXTSn1YO51GCFduyjRqQFb+noje3Bjyjv6KIAE8Ncm2 Nwi3flom6z/7FQtSokVm0X3uStqxARX67cqAokLa/g8aGBhoBFUont6A7q+be+W8 rG1FYwTIn/5jI6MDueoSD8NFGOyjNHrfGIIV3Cb2K/tFWPt7xB1aeQmnGaZBHhrp 8zBKsipVQj3ZHSKdoo0mP+uvjqklPKdD53twWngxoPydnapuUkgwG8G/gOFn1mC7 HyaGPF8dB0vv25EijZcF7uC74HQdMvPDUC+6nA35zJ1EgYpINUlyIgInXhRP9ckj Sw2aUd3yxchsaHt8BE2uLIWRbiSkOkBdVxvTna19iroLU7ivNlc5VOSKqxk7wga+ Ci4+kSFm0oZnSRz1gSqv =GoJ5 -END PGP SIGNATURE- ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 25/02/15 09:39 PM, Andrew Beekhof wrote: On 26 Feb 2015, at 8:51 am, Digimer li...@alteeve.ca wrote: On 25/02/15 04:45 PM, David Vossel wrote: - Original Message - Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea. Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine? because soon everything will run in containers. Take a look at rhel atomic and the stuff coreos is doing. The only way pacemaker will exist on those distributions is if it lives in a super privileged container. If I can respectfully disagree, or at least provide a different opinion; I'm old enough to have seen many supposed game changers come and go. Maybe containers will be different, but to simply assume so at this stage and to base the future development of HA software on that assumption seems ... risky. I think there is an argument for supporting containers, sure, but not to base plans on the assumption that they will become the be-all and end-all. Key phrase: on these distros. David's not suggesting everyone run Pacemaker like this, only people that have bought into distros that drank the container brand cool-aid (probably out of a container!). Ah, ok then. :) - -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJU7orWAAoJECChztQA3mh0fikQALV21hJnW+TeiG1YebyNHlOq la2haZcwzqUmh+sLFgYP5xgrDhbx5dDyWSvchD1nLAaDreBMDD1lD5TTgFtxPcY7 s2dyp7sEhqVsXRcQtBRdcgN9Zw1IDk6w4lQVN8ST8yxNcLOfK1/Vwk2q3kY3O/TV SQEq1uqV0FsCDvKYuWz6ReTgVAmaxkKc7wozzD4h867dlAHtvbsznkfWM81Bcbh2 U/QMd5LBeBBqYzl7cKgu9PogzBHynnZxMmKS98eZMmj5EWWA8QoSB2UPWbJE2JNj lDEl5ms87jPDcHOqnrYg1DJ69Hudj9nxE3PzS+TxnnPJvYyEghmEA6KKMAGQZzqC uAI1acxs8zV0OZ8nndwW9JsIWLPDUHiBwSXo1LyGJoHh/ke/Y/w30YeZEd2qhweB 5tfkn5H6zWB3V1i2NDCDzw7Ph95hWcHwFwbLkdekEcj0pViX+aBJlH85pmwkiNuV A4+lAjRkslRfNFZGcjV7Dj9IQMQ9sXgouNe+4mcDH0uNu+3RVjagK4HCy2IFad85 uPXTEOLNGgq08mqn1An78/AimLcBMfWZitpFGo3z00GzMCZ2b/fHIQ7d8KLJt5pt xznQZPlhtgkKYE9/PZeghXevRmRaWBGCcX3Xil2S4NAuhHx8NxqvLOuNewBBynPx o3tDOiXwfq99YpXA4zkO =GHxp -END PGP SIGNATURE- ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
- Original Message - Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea. Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine? because soon everything will run in containers. Take a look at rhel atomic and the stuff coreos is doing. The only way pacemaker will exist on those distributions is if it lives in a super privileged container. On Mon, Feb 23, 2015 at 5:20 PM, Andrew Beekhof and...@beekhof.net wrote: On 10 Feb 2015, at 1:45 pm, Serge Dubrouski serge...@gmail.com wrote: Hello Steve, Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers. And in a few years they'll work out that they need some HA features and try to retrofit them :-) In the meantime pacemaker is actually rather good at managing containers already and knows a thing or two about HA and how to bring up a complex stack of services. The one thing that would be really interesting in this area is using the our policy engine as the kubernates scheduler. So many ideas and so little time. On Sat Feb 07 2015 at 1:19:15 PM Steven Dake (stdake) std...@cisco.com wrote: Hi, I am working on Containerizing OpenStack in the Kolla project ( http://launchpad.net/kolla ). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Serge Dubrouski. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 25/02/15 04:45 PM, David Vossel wrote: - Original Message - Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea. Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine? because soon everything will run in containers. Take a look at rhel atomic and the stuff coreos is doing. The only way pacemaker will exist on those distributions is if it lives in a super privileged container. If I can respectfully disagree, or at least provide a different opinion; I'm old enough to have seen many supposed game changers come and go. Maybe containers will be different, but to simply assume so at this stage and to base the future development of HA software on that assumption seems ... risky. I think there is an argument for supporting containers, sure, but not to base plans on the assumption that they will become the be-all and end-all. - -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJU7kPSAAoJECChztQA3mh0lR4P/3JicT5jSk976BHqNcTPmWsM DHpumNORtpVPMaktC6lT6PRYeyFtYU2lqYgtBW9tpFbNai5h3qwhscV2O0mUON1L T4NGB1ZwlVAaf+ziZwxThGor/27hS29mHHVgaqR8ePhmjwxD4VX+A3f1S9QKu59B 0FuFLnWjYFMZwDV0CXGD5IxXf9aoPI47M/AwXE18Aj9TvuwblKw5z03VXUSp1kLh jFb8WcLjX+i11vXUnVnNsOHZfvAg8ejaA5NZLj4se0cMHwA0O3geAjGNAwqrACcp R0LRkxD0RBv1II3cx0Z4ZIXTSn1YO51GCFduyjRqQFb+noje3Bjyjv6KIAE8Ncm2 Nwi3flom6z/7FQtSokVm0X3uStqxARX67cqAokLa/g8aGBhoBFUont6A7q+be+W8 rG1FYwTIn/5jI6MDueoSD8NFGOyjNHrfGIIV3Cb2K/tFWPt7xB1aeQmnGaZBHhrp 8zBKsipVQj3ZHSKdoo0mP+uvjqklPKdD53twWngxoPydnapuUkgwG8G/gOFn1mC7 HyaGPF8dB0vv25EijZcF7uC74HQdMvPDUC+6nA35zJ1EgYpINUlyIgInXhRP9ckj Sw2aUd3yxchsaHt8BE2uLIWRbiSkOkBdVxvTna19iroLU7ivNlc5VOSKqxk7wga+ Ci4+kSFm0oZnSRz1gSqv =GoJ5 -END PGP SIGNATURE- ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea. Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine? On Mon, Feb 23, 2015 at 5:20 PM, Andrew Beekhof and...@beekhof.net wrote: On 10 Feb 2015, at 1:45 pm, Serge Dubrouski serge...@gmail.com wrote: Hello Steve, Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers. And in a few years they'll work out that they need some HA features and try to retrofit them :-) In the meantime pacemaker is actually rather good at managing containers already and knows a thing or two about HA and how to bring up a complex stack of services. The one thing that would be really interesting in this area is using the our policy engine as the kubernates scheduler. So many ideas and so little time. On Sat Feb 07 2015 at 1:19:15 PM Steven Dake (stdake) std...@cisco.com wrote: Hi, I am working on Containerizing OpenStack in the Kolla project ( http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Serge Dubrouski. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
- Original Message - Hi, Hey Steve, Good to see you around :) I am working on Containerizing OpenStack in the Kolla project ( http://launchpad.net/kolla ). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. Knowing what I know about OpenStack HA now, that is a bad choice. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Yep, this is the way to do it. My plan is to have pacemaker running in a container, and have pacemaker capable of launching resources within containers. We already have a Docker resource agent. You can find it here, https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker Using that agent, pacemaker can launch a docker container, and then monitor the container by performing health checks within the container. Here's an example of how I'm using this technique to manage a containerized apache instance. https://github.com/davidvossel/phd/blob/master/scenarios/docker-apache-ap.scenario#L96 Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their I actually already got pacemaker+corosync running in a container for testing purposes. If you're interested you can checkout some of that work here, https://github.com/davidvossel/phd/tree/master/lib/docker . The phd_docker_utils.sh file holds most of the interesting parts. normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging yes, and from there kolla-mgr hands the containers off to pacemaker to manage. kolla is the orchestration, pacemaker is the scheduler for performing those tasks. This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
On 10 Feb 2015, at 1:45 pm, Serge Dubrouski serge...@gmail.com wrote: Hello Steve, Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers. And in a few years they'll work out that they need some HA features and try to retrofit them :-) In the meantime pacemaker is actually rather good at managing containers already and knows a thing or two about HA and how to bring up a complex stack of services. The one thing that would be really interesting in this area is using the our policy engine as the kubernates scheduler. So many ideas and so little time. On Sat Feb 07 2015 at 1:19:15 PM Steven Dake (stdake) std...@cisco.com wrote: Hi, I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
On 8 Feb 2015, at 7:09 am, Steven Dake (stdake) std...@cisco.com wrote: Hi, I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. systemctl or pacemaker-remote in stand-alone mode? I can imagine the latter more so than the former. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. Separate to even pacemaker? My container-foo is weak... does they allow traditional IPC mechanisms between containers? These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? If you wrote an RA that did so, then it can do whatever you like. However the concept seems prone to internal split-brain given the number of entities wanting a piece of the container. Will the containers always be in the pacemaker config or only added when someone runs 'deploy'? How will pacemaker know when to start monitoring? What should pacemaker do if it detects the container as failed? How can it tell the difference between a failure and 'kolla-mgr kill messaging'? Could pacemaker be the 3rd party that does the starting? If so, perhaps look at the Docker agent. If not, you will need to make sure pacemaker doesn't start health checking the container until the 3rd party has completely started it. I'm not saying it cant work as described, I just want to understand the proposal better before I go too much further. Or would we be better off with some systemd integration with kolla-mgr? systemd integration is rarely the answer :-) Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
Hello Steve, Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers. On Sat Feb 07 2015 at 1:19:15 PM Steven Dake (stdake) std...@cisco.com wrote: Hi, I am working on Containerizing OpenStack in the Kolla project ( http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?
Hi, I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound. There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla. Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software. An example: Lets say a rabbitmq container wants to run: The user would run kolla-mgr deploy messaging This would run a small bit of code to launch the docker container set for messaging. Could pacemaker run something like Kolla-mgr status messaging To control the lifecycle of the processes? Or would we be better off with some systemd integration with kolla-mgr? Thoughts welcome Regards, -steve ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
