Re: [Linux-HA] Multiple instances of heartbeat
On 03/15/2013 01:20 PM, GGS (linux ha) wrote: > Virtualization has a huge penalty on performance, specially > at the IO level. At another place we do Xen and KVM with up to > 40 VMs/server and when there is any kind of IO (disk specially) going > on things slow down to a crawl. I'm yet to find anything that can deal with i/o. I recently spent a couple of weeks poking at ceph, it doesn't live up to the sales brochure either... I expect if you can roll out a dedicated 10GBe network for your iscsis you might get usable i/o speeds. :( -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Fri, 2013-03-15 at 13:08 -0500, Dimitri Maziuk wrote: > On 03/15/2013 12:59 PM, GGS (linux ha) wrote: > > > Unfortunately I'm not at liberty to discuss the full architecture > > or what they are doing without written permission, which would > > make it clear why we are going the path we are. > > Yeah, I suspected something like that. Hopefully I won't ever need to > know. ;-) Actually is not serious like that, it's more of a legal, standard corporate structure. It would be better if they made it available for public discussion so that ideas could fly more freely. You guys could probably see something simple that we are missing. Oh well... that's a whole other tangent. > > (I'd still argue that a full vm solution should have less maintenance > overhead in the long run -- or at least it looks that way now.) Virtualization has a huge penalty on performance, specially at the IO level. At another place we do Xen and KVM with up to 40 VMs/server and when there is any kind of IO (disk specially) going on things slow down to a crawl. Once I learn pacemaker I think things will be much better :-) Alberto ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 03/15/2013 12:59 PM, GGS (linux ha) wrote: > Unfortunately I'm not at liberty to discuss the full architecture > or what they are doing without written permission, which would > make it clear why we are going the path we are. Yeah, I suspected something like that. Hopefully I won't ever need to know. ;-) (I'd still argue that a full vm solution should have less maintenance overhead in the long run -- or at least it looks that way now.) -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Fri, 2013-03-15 at 12:32 -0500, Dimitri Maziuk wrote: > On 03/15/2013 11:55 AM, Lars Marowsky-Bree wrote: > ... > > Right. Thankfully, we already have that, it's called pacemaker ;-) > > Which brings me back to my original problem with the concept: I can > think of only one reason to "failover" services (as opposed to > hardware), and that is your daemons are crashing all the time during > normal operation. If I needed a solution for that, HA would be fairly > low on my list of things to look at. You need to look back at the original description I gave. These are not your typical web stack or back office apps. We do have them running for years without failures/crashes, but in the 24x7 environment they are in we try to minimize the risk of downtime regardless. Unfortunately I'm not at liberty to discuss the full architecture or what they are doing without written permission, which would make it clear why we are going the path we are. Alberto ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Fri, 2013-03-15 at 17:55 +0100, Lars Marowsky-Bree wrote: > Yeah, I'd agree that today there are scenarios where "a cloud" makes > more sense then a traditional HA environment. OpenStack et al still have > to up their HA game a bit, though. You are being way too kind, a lot of improvement is needed. > > > > And the concept of HA clusters predates "the cloud" slightly. > > Relevant if you're looking at maintenance/upgrade on an existing > > cluster. Patching heartbeat to manage 200 services independently sounds > > like a new project. > > Right. Thankfully, we already have that, it's called pacemaker ;-) And that's where I'm looking next, I hope it is "easy". > > Regards, > Lars Alberto ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Fri, 2013-03-15 at 11:43 -0500, Dimitri Maziuk wrote: > On 03/15/2013 10:08 AM, Lars Marowsky-Bree wrote: > > > You're contradicting yourself ;-) Pacemaker in fact gives you the > > management you suggest for the "cloud" use case - whether the > services > > are handled natively or encapsulated into a VM. > > Yeah, I suppose. I meant going Open/CloudStack. > (We get to write buzzword-compliant funding proposals, or I don't get > to > eat. So my perspective is skewed towards the hottest shiny du jour...) These projects do not relate well to full VMs, so it is actually not a good direction. But yes, we do use the load balancer and VMs approach for other things, so I am familiar with that type of architecture. > > > And the concept of HA clusters predates "the cloud" slightly. > > Relevant if you're looking at maintenance/upgrade on an existing > cluster. Patching heartbeat to manage 200 services independently > sounds > like a new project. The current solution was written in-house. We are looking to replace it. Based on the info from the list heartbeat is out, so I'll look more into pacemaker. I know what you mean about the buzzwords. I'm trying to avoid them. :-) Alberto ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 03/15/2013 11:55 AM, Lars Marowsky-Bree wrote: ... > Right. Thankfully, we already have that, it's called pacemaker ;-) Which brings me back to my original problem with the concept: I can think of only one reason to "failover" services (as opposed to hardware), and that is your daemons are crashing all the time during normal operation. If I needed a solution for that, HA would be fairly low on my list of things to look at. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 2013-03-15T11:43:56, Dimitri Maziuk wrote: > Yeah, I suppose. I meant going Open/CloudStack. > (We get to write buzzword-compliant funding proposals, or I don't get to > eat. So my perspective is skewed towards the hottest shiny du jour...) Yeah, I'd agree that today there are scenarios where "a cloud" makes more sense then a traditional HA environment. OpenStack et al still have to up their HA game a bit, though. > > And the concept of HA clusters predates "the cloud" slightly. > Relevant if you're looking at maintenance/upgrade on an existing > cluster. Patching heartbeat to manage 200 services independently sounds > like a new project. Right. Thankfully, we already have that, it's called pacemaker ;-) Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 03/15/2013 10:08 AM, Lars Marowsky-Bree wrote: > You're contradicting yourself ;-) Pacemaker in fact gives you the > management you suggest for the "cloud" use case - whether the services > are handled natively or encapsulated into a VM. Yeah, I suppose. I meant going Open/CloudStack. (We get to write buzzword-compliant funding proposals, or I don't get to eat. So my perspective is skewed towards the hottest shiny du jour...) > And the concept of HA clusters predates "the cloud" slightly. Relevant if you're looking at maintenance/upgrade on an existing cluster. Patching heartbeat to manage 200 services independently sounds like a new project. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 2013-03-15T09:54:22, Dimitri Maziuk wrote: > I've always had difficulties with the concept: the way I see it if your > hardware fails you want *all* your 200+ services moved. If you want them > independently moved to different places, you're likely better off with a > full cloud solution. If you want them moved while hardware's still up > you're probably looking for load balancing, not HA. > > I'm sure you can patch heartbeat to replace all hardcoded stuff with > config file settings. Or use pacemaker's ability to manage service > groups more or less independently. I'm not sure why you'd want to use > either that way. You're contradicting yourself ;-) Pacemaker in fact gives you the management you suggest for the "cloud" use case - whether the services are handled natively or encapsulated into a VM. And the concept of HA clusters predates "the cloud" slightly. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 3/14/2013 11:15 AM, Alberto Alonso wrote: > That's what I thought. The emails from 2009 seemed to indicate > that it was possible to run multiple instances. I've always had difficulties with the concept: the way I see it if your hardware fails you want *all* your 200+ services moved. If you want them independently moved to different places, you're likely better off with a full cloud solution. If you want them moved while hardware's still up you're probably looking for load balancing, not HA. I'm sure you can patch heartbeat to replace all hardcoded stuff with config file settings. Or use pacemaker's ability to manage service groups more or less independently. I'm not sure why you'd want to use either that way. Dima ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
hi, On Thu, Mar 14, 2013 at 11:15:29AM -0500, Alberto Alonso wrote: > On Thu, 2013-03-14 at 16:26 +0100, Lars Marowsky-Bree wrote: > > On 2013-03-14T09:44:11, "GGS (linux ha)" wrote: > > > > That's fine. But the cluster software really assumes that only one > > instance of it is running per server - said instance can then manage > > multiple software stacks, though. > > Got it. That's what I was asking. > > > > No. Pacemaker allows resources and groups (probably the equivalent of > > your "stacks") to be individually managed. > > > > If you want to bring down pacemaker itself for maintenance, you'd detach > > via maintenance mode, stop, update, restart, reattach. > > I'll have to dig in deeper, it may be a possibility. We really > would like to move away from the in-house built solution. > > > > But there is a point where this matters, namely IO fencing/STONITH. In > > case of a real server failure, you don't want 200+ independent fencing > > processes to trigger. > > Believe it or not, I would actually rather have the 200+ > fencing processes to trigger. But that is not a > requirement. I just need to ensure failover completes > within the allowed time. > > > > > Yes. That's called multitasking/virtualization/cloud. We get that. ;-) > > Multitasking yes, virtualization no, that's another discussion :-) > > > > But just like you only have "one kernel" per physical server, you also > > only have one cluster stack that then manages multiple stacks. We even > > got ACLs so that you can grant people access to only the bits they're > > allowed to manage, etc. > > > > What you plan - running multiple heartbeat v1 setups on one node - will > > not work reliably. Running multiple pacemaker instances per node/OS > > image will not work either. > > That's what I thought. The emails from 2009 seemed to indicate > that it was possible to run multiple instances. > > I asked because I suspected that it really wasn't the case. Thanks > for confirming it. I'll dig deeper into pacemaker and see how I > can make it work for our use case. > > One quick question on pacemaker. If I add a new stack, do I need > to bring the old ones down (or fail them) to add it to pacemaker? No. Thanks, Dejan > >From your comment above it seems that I wouldn't, but I just want > to make sure. > > Thanks, > > Alberto > > > > > > > > Regards, > > Lars > > > -- > Alberto AlonsoGlobal Gate Systems LLC. > (512) 351-7233http://www.ggsys.net > Monitoring the metrics that are important to you in real time > > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Thu, 2013-03-14 at 16:26 +0100, Lars Marowsky-Bree wrote: > On 2013-03-14T09:44:11, "GGS (linux ha)" wrote: > > That's fine. But the cluster software really assumes that only one > instance of it is running per server - said instance can then manage > multiple software stacks, though. Got it. That's what I was asking. > > No. Pacemaker allows resources and groups (probably the equivalent of > your "stacks") to be individually managed. > > If you want to bring down pacemaker itself for maintenance, you'd detach > via maintenance mode, stop, update, restart, reattach. I'll have to dig in deeper, it may be a possibility. We really would like to move away from the in-house built solution. > > But there is a point where this matters, namely IO fencing/STONITH. In > case of a real server failure, you don't want 200+ independent fencing > processes to trigger. Believe it or not, I would actually rather have the 200+ fencing processes to trigger. But that is not a requirement. I just need to ensure failover completes within the allowed time. > > Yes. That's called multitasking/virtualization/cloud. We get that. ;-) Multitasking yes, virtualization no, that's another discussion :-) > > But just like you only have "one kernel" per physical server, you also > only have one cluster stack that then manages multiple stacks. We even > got ACLs so that you can grant people access to only the bits they're > allowed to manage, etc. > > What you plan - running multiple heartbeat v1 setups on one node - will > not work reliably. Running multiple pacemaker instances per node/OS > image will not work either. That's what I thought. The emails from 2009 seemed to indicate that it was possible to run multiple instances. I asked because I suspected that it really wasn't the case. Thanks for confirming it. I'll dig deeper into pacemaker and see how I can make it work for our use case. One quick question on pacemaker. If I add a new stack, do I need to bring the old ones down (or fail them) to add it to pacemaker? >From your comment above it seems that I wouldn't, but I just want to make sure. Thanks, Alberto > > > Regards, > Lars > -- Alberto AlonsoGlobal Gate Systems LLC. (512) 351-7233http://www.ggsys.net Monitoring the metrics that are important to you in real time ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 2013-03-14T09:44:11, "GGS (linux ha)" wrote: > That's the problem. We do not run a cluster > of servers. Our logical unit is the software > stack. (see below) That's fine. But the cluster software really assumes that only one instance of it is running per server - said instance can then manage multiple software stacks, though. > > You see to want to shoot yourself in the foot. Why? > Most people think the unit is the server. We > have servers capable of running 50-100 > stacks. Our unit is the stack, which has a > direct connection to the physical world (each > controls specific equipment manned by specific > people). Each stack is subject to its own > rules, resources and downtime. As such we bring > them up and down, do updates and configuration > changes individually. That's fine. And one of the reasons VMs were invented. > If we were to have a single daemon in charge > of all the stacks any maintenance would need > to be coordinated across all the stacks. Bringing > down pacemaker would affect everyone. In our 24x7 > operation there is just no overlap. No. Pacemaker allows resources and groups (probably the equivalent of your "stacks") to be individually managed. If you want to bring down pacemaker itself for maintenance, you'd detach via maintenance mode, stop, update, restart, reattach. But there is a point where this matters, namely IO fencing/STONITH. In case of a real server failure, you don't want 200+ independent fencing processes to trigger. > There is nothing inherently complex with the > stack model, it is just not as common as the > server model. I always laugh when people talk > about having hundreds or thousands of servers, because > switching to a stack model and proper utilization > of hardware resources can save a ton of money. Yes. That's called multitasking/virtualization/cloud. We get that. ;-) But just like you only have "one kernel" per physical server, you also only have one cluster stack that then manages multiple stacks. We even got ACLs so that you can grant people access to only the bits they're allowed to manage, etc. What you plan - running multiple heartbeat v1 setups on one node - will not work reliably. Running multiple pacemaker instances per node/OS image will not work either. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Thu, 2013-03-14 at 09:53 -0500, Dimitri Maziuk wrote: > On 3/14/2013 9:44 AM, GGS (linux ha) wrote: > > > ... I always laugh when people talk > > about having hundreds or thousands of servers, because > > switching to a stack model and proper utilization > > of hardware resources can save a ton of money. > > The flip side is when caps on the mobo go dry you lose 50-100 > services/stacks instead of just one. That's the beauty of HA, they all switch and hopefully only minor inconveniences. Of course on high end server hardware this is rather rare in the first place. Alberto > > Dima > > > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 3/14/2013 9:44 AM, GGS (linux ha) wrote: > ... I always laugh when people talk > about having hundreds or thousands of servers, because > switching to a stack model and proper utilization > of hardware resources can save a ton of money. The flip side is when caps on the mobo go dry you lose 50-100 services/stacks instead of just one. Dima ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On Thu, 2013-03-14 at 09:55 +0100, Lars Marowsky-Bree wrote: > On 2013-03-13T20:13:30, "GGS (linux ha)" > wrote: > > > Their configuration, resources, etc. are > > not intermingled, so we prefer not to > > configure them in a single setup. For > > simplicity we are using the R1 built > > into heartbeat and not CRM. > > What you want to do is very simple to do with a pacemaker based cluster > and very difficult to get right with multiple instances of heartbeat > (among other things: fencing, getting multiple instances to run in the > first place, etc). That's the problem. We do not run a cluster of servers. Our logical unit is the software stack. (see below) > > > I don't mind the administrative overhead of > > putting a whole ha tree an compile a copy > > for each of my stacks. If that's what it > > takes that's what I'll do. > > So, to achieve simplicity, you're willing to pay a significant overhead > on complexity. That makes sense. There is nothing complex about keeping a copy of the software per stack. We are already doing that. Basically the common software runs as many instances as there are stacks and just uses different configuration files for each. > > You see to want to shoot yourself in the foot. Why? Most people think the unit is the server. We have servers capable of running 50-100 stacks. Our unit is the stack, which has a direct connection to the physical world (each controls specific equipment manned by specific people). Each stack is subject to its own rules, resources and downtime. As such we bring them up and down, do updates and configuration changes individually. If we were to have a single daemon in charge of all the stacks any maintenance would need to be coordinated across all the stacks. Bringing down pacemaker would affect everyone. In our 24x7 operation there is just no overlap. What I can't get is why heartbeat has things/paths hard coded. The norm is to be able to specify configuration options from the command line. Something like --cfg filename There is nothing inherently complex with the stack model, it is just not as common as the server model. I always laugh when people talk about having hundreds or thousands of servers, because switching to a stack model and proper utilization of hardware resources can save a ton of money. Thanks, Alberto > > > Regards, > Lars ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple instances of heartbeat
On 2013-03-13T20:13:30, "GGS (linux ha)" wrote: > Their configuration, resources, etc. are > not intermingled, so we prefer not to > configure them in a single setup. For > simplicity we are using the R1 built > into heartbeat and not CRM. What you want to do is very simple to do with a pacemaker based cluster and very difficult to get right with multiple instances of heartbeat (among other things: fencing, getting multiple instances to run in the first place, etc). > I don't mind the administrative overhead of > putting a whole ha tree an compile a copy > for each of my stacks. If that's what it > takes that's what I'll do. So, to achieve simplicity, you're willing to pay a significant overhead on complexity. That makes sense. You see to want to shoot yourself in the foot. Why? Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
