Re: [openstack-dev] [FUEL] Zabbix in HA mode
Hi Andrew and all! On 05 Jan 2015, at 22:05, Andrew Woodward xar...@gmail.com wrote: On Tue, Nov 25, 2014 at 5:21 AM, Bartosz Kupidura bkupid...@mirantis.com wrote: Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. This needs to be discouraged as much as putting mongo-db on the controllers. We know that, and we can use UI warning for that. For the reasons Mike provided our users need it. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). Again, not forced on controllers, this is very bad. Controllers: While there is development use cases to deploy monitoring on combined controllers, and it can make use of the already existing pacemaker cluster, this is the wrong direction to point users. There are many reasons this is bad: for one, monitoring can become quite loaded, and as we've seen secondary load on the controllers can collapse the entire control plane. Secondly running monitoring on the cluster may also result in the monitoring going offline if the cluster does, from my own experience, not being able to see your monitoring is nearly worse than having everything down and leads to lost precious moments of downtime SLA. HA Scaling: Just like with controllers, our other HA components need to support a scale of 1 to N. This is important as a cluster will need to scale, or as the operator moves from POC to Production, they can deploy more hardware. This also helps alleviate some of the not enough nodes issues mentioned in the thread already Your concenrs are 100% valid and I agree with them. But what about small instalaltions, where only 4 physical machines are available? We are already wasting one for Fuel node, and 3 for controllers. There is hardware with similar setup and it seems to be very popular. This is what we are trying to address. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
On 01/05/2015 04:05 PM, Andrew Woodward wrote: snip Secondly running monitoring on the cluster may also result in the monitoring going offline if the cluster does, from my own experience, not being able to see your monitoring is nearly worse than having everything down and leads to lost precious moments of downtime SLA. Yep, this is exactly my experience as well. Excellent point. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
Rob, This is of limited value to my business due to the GPL license -- so my company's lawyers tell me. I will be unable to take advantage of what looks to be a solid solution from what I can see of Zabbix. Are there any risks to Fuel (open source contamination) from this approach? I doubt it but I want to make sure you are considering this. Zabbix is GPL 2.0, however the impact of this license is only when developing against its source. Using a GPL program through its standard interfaces does not pull in any of the license requirements that your legal team may be upset with. (AGPL is different story). Also GPL programs are used throughout the base Linux operating system. In some cases we may modify packages (including GPL ones), their source is provided and can be found at [1] In the scope of fuel, we configure Zabbix through standard interfaces with our puppet manifests which are Apache 2.0 [2] [1] https://review.fuel-infra.org/ [2] https://github.com/stackforge/fuel-library/tree/master/deployment/puppet/zabbix On Tue, Nov 25, 2014 at 6:07 AM, Rob Basham rob...@us.ibm.com wrote: Rob Basham Cloud Systems Software Architecture 971-344-1999 Bartosz Kupidura bkupid...@mirantis.com wrote on 11/25/2014 05:21:59 AM: From: Bartosz Kupidura bkupid...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 11/25/2014 05:26 AM Subject: [openstack-dev] [FUEL] Zabbix in HA mode Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). In next stage, we can allow users to deploy zabbix-server on dedicated node OR on controllers for performance reasons. But for now we should force zabbix-server to be deployed on controllers. BP is in initial phase, but code is ready and working with Fuel 5.1. Now im checking if it works with master. Any comments are welcome! This is of limited value to my business due to the GPL license -- so my company's lawyers tell me. I will be unable to take advantage of what looks to be a solid solution from what I can see of Zabbix. Are there any risks to Fuel (open source contamination) from this approach? I doubt it but I want to make sure you are considering this. BP link: https://blueprints.launchpad.net/fuel/+spec/zabbix-ha Best Regards, Bartosz Kupidura ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Andrew Mirantis Ceph community ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
On Tue, Nov 25, 2014 at 5:21 AM, Bartosz Kupidura bkupid...@mirantis.com wrote: Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. This needs to be discouraged as much as putting mongo-db on the controllers. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. No, this must remain a separate role and can not be forced onto the controllers the user should be discouraged from doing this. The corosync code is quickly becoming granular enough to stand up a CRM cluster elsewhere. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. Seems a good way to handle it, but would it run well for a plugin that wants to be monitored (since they run after) When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). Again, not forced on controllers, this is very bad. Controllers: While there is development use cases to deploy monitoring on combined controllers, and it can make use of the already existing pacemaker cluster, this is the wrong direction to point users. There are many reasons this is bad: for one, monitoring can become quite loaded, and as we've seen secondary load on the controllers can collapse the entire control plane. Secondly running monitoring on the cluster may also result in the monitoring going offline if the cluster does, from my own experience, not being able to see your monitoring is nearly worse than having everything down and leads to lost precious moments of downtime SLA. HA Scaling: Just like with controllers, our other HA components need to support a scale of 1 to N. This is important as a cluster will need to scale, or as the operator moves from POC to Production, they can deploy more hardware. This also helps alleviate some of the not enough nodes issues mentioned in the thread already -- Andrew Mirantis Ceph community ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
For small installs we still have to consider an option of roles combination, and placing Zabbix on controllers. Fuel disk allocation logic should be smart and has to allocate separate disk for it where possible. On Wednesday, November 26, 2014, Stanislaw Bogatkin sbogat...@mirantis.com wrote: Hi Bartosz, As for me - zabbix is bad practice to place on controller nodes when large installations will be monitoring, cause it can slow down disk IO on big db. If it happen - controllers can became unresponsible to other services on controllers. Zabbix guys recommended to use separate station for large installations. But zabbix in HA on dedicated nodes (not on the controllers) is very good idea to eliminate monitoring SPOF. On Tue, Nov 25, 2014 at 8:07 PM, Mike Scherbakov mscherba...@mirantis.com javascript:_e(%7B%7D,'cvml','mscherba...@mirantis.com'); wrote: Regarding the licensing, it should not be an issue because we provide all source code (if not as git repos, then as source RPMs/DEBs). On Tue, Nov 25, 2014 at 7:34 PM, Bartosz Kupidura bkupid...@mirantis.com javascript:_e(%7B%7D,'cvml','bkupid...@mirantis.com'); wrote: Hello Vladimir, I agree. But in most cases, zabbix-server would be moved from failed node by pacemaker. Moreover some clients dont want to „waste” 3 additional servers only for monitoring. As i said, this is only first drop of zabbix HA. Later we can allow user to deploy zabbix-server not only on controllers, but also on dedicated nodes. Best Regards, Bartosz Kupidura Wiadomość napisana przez Vladimir Kuklin vkuk...@mirantis.com javascript:_e(%7B%7D,'cvml','vkuk...@mirantis.com'); w dniu 25 lis 2014, o godz. 15:47: Bartosz, It is obviously possible to install zabbix on the master nodes and put it under pacemaker control. But it seems very strange for me to monitor something with software located on the nodes that you are monitoring. On Tue, Nov 25, 2014 at 4:21 PM, Bartosz Kupidura bkupid...@mirantis.com javascript:_e(%7B%7D,'cvml','bkupid...@mirantis.com'); wrote: Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). In next stage, we can allow users to deploy zabbix-server on dedicated node OR on controllers for performance reasons. But for now we should force zabbix-server to be deployed on controllers. BP is in initial phase, but code is ready and working with Fuel 5.1. Now im checking if it works with master. Any comments are welcome! BP link: https://blueprints.launchpad.net/fuel/+spec/zabbix-ha Best Regards, Bartosz Kupidura ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org'); http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com www.mirantis.ru vkuk...@mirantis.com javascript:_e(%7B%7D,'cvml','vkuk...@mirantis.com'); ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org'); http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org'); http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org'); http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. But zabbix-server will stay and user will be able to assign this role where he wants? If so there will be no limitations on roles allocation strategy that user can use for cluster Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically It must not be in orchestrator (i guess you are talking about serializer) by some cluster attribute or another hack. I thought about such kind of role placement during granular deployment design, and it can be done in a next way: Zabbix-monitoring (i like zabbix-agent more) to all servers if zabbix-server is added to cluster, and then operator should be able to remove zabbix-monitoring from some nodes. But more importantly he will be able to see roles to nodes placement in a very explicit manner ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [FUEL] Zabbix in HA mode
Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). In next stage, we can allow users to deploy zabbix-server on dedicated node OR on controllers for performance reasons. But for now we should force zabbix-server to be deployed on controllers. BP is in initial phase, but code is ready and working with Fuel 5.1. Now im checking if it works with master. Any comments are welcome! BP link: https://blueprints.launchpad.net/fuel/+spec/zabbix-ha Best Regards, Bartosz Kupidura ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
Rob Basham Cloud Systems Software Architecture 971-344-1999 Bartosz Kupidura bkupid...@mirantis.com wrote on 11/25/2014 05:21:59 AM: From: Bartosz Kupidura bkupid...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 11/25/2014 05:26 AM Subject: [openstack-dev] [FUEL] Zabbix in HA mode Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). In next stage, we can allow users to deploy zabbix-server on dedicated node OR on controllers for performance reasons. But for now we should force zabbix-server to be deployed on controllers. BP is in initial phase, but code is ready and working with Fuel 5.1. Now im checking if it works with master. Any comments are welcome! This is of limited value to my business due to the GPL license -- so my company's lawyers tell me. I will be unable to take advantage of what looks to be a solid solution from what I can see of Zabbix. Are there any risks to Fuel (open source contamination) from this approach? I doubt it but I want to make sure you are considering this. BP link: https://blueprints.launchpad.net/fuel/+spec/zabbix-ha Best Regards, Bartosz Kupidura ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
Bartosz, It is obviously possible to install zabbix on the master nodes and put it under pacemaker control. But it seems very strange for me to monitor something with software located on the nodes that you are monitoring. On Tue, Nov 25, 2014 at 4:21 PM, Bartosz Kupidura bkupid...@mirantis.com wrote: Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). In next stage, we can allow users to deploy zabbix-server on dedicated node OR on controllers for performance reasons. But for now we should force zabbix-server to be deployed on controllers. BP is in initial phase, but code is ready and working with Fuel 5.1. Now im checking if it works with master. Any comments are welcome! BP link: https://blueprints.launchpad.net/fuel/+spec/zabbix-ha Best Regards, Bartosz Kupidura ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com http://www.mirantis.ru/ www.mirantis.ru vkuk...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [FUEL] Zabbix in HA mode
Hello Vladimir, I agree. But in most cases, zabbix-server would be moved from failed node by pacemaker. Moreover some clients dont want to „waste” 3 additional servers only for monitoring. As i said, this is only first drop of zabbix HA. Later we can allow user to deploy zabbix-server not only on controllers, but also on dedicated nodes. Best Regards, Bartosz Kupidura Wiadomość napisana przez Vladimir Kuklin vkuk...@mirantis.com w dniu 25 lis 2014, o godz. 15:47: Bartosz, It is obviously possible to install zabbix on the master nodes and put it under pacemaker control. But it seems very strange for me to monitor something with software located on the nodes that you are monitoring. On Tue, Nov 25, 2014 at 4:21 PM, Bartosz Kupidura bkupid...@mirantis.com wrote: Hello All, Im working on Zabbix implementation which include HA support. Zabbix server should be deployed on all controllers in HA mode. Currently we have dedicated role 'zabbix-server', which does not support more than one zabbix-server. Instead of this we will move monitoring solution (zabbix), as an additional component. We will introduce additional role 'zabbix-monitoring', assigned to all servers with lowest priority in serializer (run puppet after every other roles) when zabbix is enabled. 'Zabbix-monitoring' role will be assigned automatically. When zabbix component is enabled, we will install zabbix-server on all controllers in active-backup mode (pacemaker+haproxy). In next stage, we can allow users to deploy zabbix-server on dedicated node OR on controllers for performance reasons. But for now we should force zabbix-server to be deployed on controllers. BP is in initial phase, but code is ready and working with Fuel 5.1. Now im checking if it works with master. Any comments are welcome! BP link: https://blueprints.launchpad.net/fuel/+spec/zabbix-ha Best Regards, Bartosz Kupidura ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com www.mirantis.ru vkuk...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
