Re: Tuning HAProxy on EC2 instances?
It seems that Ec2 is too elastic. Also it seems to be more targeted by hackers than real hosting solutions. One solution to overcome this partially is to reserve some IP's so changes that they were under attack before are low. As for the performance problems there is no way to control the throughput. In EU zone I could get around 70 MB/s but unstable (not speaking about haproxy). From: Brent Walker To: haproxy@formilux.org; Alexander Staubo Sent: Mon, February 1, 2010 12:09:33 PM Subject: Re: Tuning HAProxy on EC2 instances? We are doing about the same amount of traffic on a CentOS AMI but using a small instance. Has been problem free for better than 6 months of use. On Mon, Feb 1, 2010 at 1:19 AM, Joe Williams wrote: >>We use haproxy and EC2 instances as load balancers for our clusters. The >>tuning we use is pretty standard (somaxconn, nf_conntrack_max, >>tcp_fin_timeout, rmem_max, wmem_max and etc) running vanilla ubuntu AMIs. >>While EC2's instances and network have performance problems it is possible to >>get reasonable reliability and performance from them. We push between 10s of >>Mbps through a single c1.medium without issues, not sure about beyond that. > >>-Joe > > > > >>On 1/31/10 3:14 PM, Willy Tarreau wrote: > >>>Hi Alexander, >> >>>>On Sun, Jan 31, 2010 at 11:36:02PM +0100, Alexander Staubo wrote: >>>> >> >>>>>Has anyone any experience tuning HAProxy for performance when running >>>>>>on Amazon EC2 instances? For example, are there any kernel parameters >>>>>>that should be tuned differently, or are some instance types better >>>>>>than others? Does HAProxy generally perform well on EC2? >>>>>> >>> >>well, last year I helped some guys in charge of a world wide sports >>>>event which was hosted there. The performance was terrible. Completely >>>>unstable. It was impossible to tune anything. Ping times would vary a >>>>lot. It was impossible to know where the bottlenecks were, because >>>>every machine was showing limited performance in turn without >>>>necessarily having its CPU saturated. It was noticed that the internal >>>>network was at least faulty, because the observed network congestions >>>>were not constat and moving between machines. Sometimes it was even >>>>almost impossible to type in SSH. We also discovered that when they >>>>bought new nodes, some of them were under massive attacks, most likely >>>>because people who are attacked quickly drop the nodes with the IPs >>>>that belong to them and create new ones. So the attacked ones will >>>>be picked by the next customer... Finally they moved to a real hosting >>>>company with real machines and real performance in order to be able >>>>to participate at least to a little part of the event. >> >>>>In this experience, I think that for them, everything was virtual : >>>>the machines, the network, the support, the availability, the visitors >>>>and finally the profit. >> >>>>I really can't say what you could play on to improve quality. After >>>>having spent 3 full nights working with them on their machine, no >>>>sensible trend appeared whatever we did. I think the real knobs are >>>>outside your scope, on the other side of the VM :-/ >> >>>>Regards, >>>>Willy >> >> >>>> >> >>-- >>Name: Joseph A. Williams >>Email: j...@joetify.com >>Blog: http://www.joeandmotorboat.com/ > > >
Re: Tuning HAProxy on EC2 instances?
We are doing about the same amount of traffic on a CentOS AMI but using a small instance. Has been problem free for better than 6 months of use. On Mon, Feb 1, 2010 at 1:19 AM, Joe Williams wrote: > > We use haproxy and EC2 instances as load balancers for our clusters. The > tuning we use is pretty standard (somaxconn, nf_conntrack_max, > tcp_fin_timeout, rmem_max, wmem_max and etc) running vanilla ubuntu AMIs. > While EC2's instances and network have performance problems it is possible > to get reasonable reliability and performance from them. We push between 10s > of Mbps through a single c1.medium without issues, not sure about beyond > that. > > -Joe > > > > > On 1/31/10 3:14 PM, Willy Tarreau wrote: > >> Hi Alexander, >> >> On Sun, Jan 31, 2010 at 11:36:02PM +0100, Alexander Staubo wrote: >> >> >>> Has anyone any experience tuning HAProxy for performance when running >>> on Amazon EC2 instances? For example, are there any kernel parameters >>> that should be tuned differently, or are some instance types better >>> than others? Does HAProxy generally perform well on EC2? >>> >>> >> well, last year I helped some guys in charge of a world wide sports >> event which was hosted there. The performance was terrible. Completely >> unstable. It was impossible to tune anything. Ping times would vary a >> lot. It was impossible to know where the bottlenecks were, because >> every machine was showing limited performance in turn without >> necessarily having its CPU saturated. It was noticed that the internal >> network was at least faulty, because the observed network congestions >> were not constat and moving between machines. Sometimes it was even >> almost impossible to type in SSH. We also discovered that when they >> bought new nodes, some of them were under massive attacks, most likely >> because people who are attacked quickly drop the nodes with the IPs >> that belong to them and create new ones. So the attacked ones will >> be picked by the next customer... Finally they moved to a real hosting >> company with real machines and real performance in order to be able >> to participate at least to a little part of the event. >> >> In this experience, I think that for them, everything was virtual : >> the machines, the network, the support, the availability, the visitors >> and finally the profit. >> >> I really can't say what you could play on to improve quality. After >> having spent 3 full nights working with them on their machine, no >> sensible trend appeared whatever we did. I think the real knobs are >> outside your scope, on the other side of the VM :-/ >> >> Regards, >> Willy >> >> >> >> > > -- > Name: Joseph A. Williams > Email: j...@joetify.com > Blog: http://www.joeandmotorboat.com/ > > >
Re: Tuning HAProxy on EC2 instances?
We use haproxy and EC2 instances as load balancers for our clusters. The tuning we use is pretty standard (somaxconn, nf_conntrack_max, tcp_fin_timeout, rmem_max, wmem_max and etc) running vanilla ubuntu AMIs. While EC2's instances and network have performance problems it is possible to get reasonable reliability and performance from them. We push between 10s of Mbps through a single c1.medium without issues, not sure about beyond that. -Joe On 1/31/10 3:14 PM, Willy Tarreau wrote: Hi Alexander, On Sun, Jan 31, 2010 at 11:36:02PM +0100, Alexander Staubo wrote: Has anyone any experience tuning HAProxy for performance when running on Amazon EC2 instances? For example, are there any kernel parameters that should be tuned differently, or are some instance types better than others? Does HAProxy generally perform well on EC2? well, last year I helped some guys in charge of a world wide sports event which was hosted there. The performance was terrible. Completely unstable. It was impossible to tune anything. Ping times would vary a lot. It was impossible to know where the bottlenecks were, because every machine was showing limited performance in turn without necessarily having its CPU saturated. It was noticed that the internal network was at least faulty, because the observed network congestions were not constat and moving between machines. Sometimes it was even almost impossible to type in SSH. We also discovered that when they bought new nodes, some of them were under massive attacks, most likely because people who are attacked quickly drop the nodes with the IPs that belong to them and create new ones. So the attacked ones will be picked by the next customer... Finally they moved to a real hosting company with real machines and real performance in order to be able to participate at least to a little part of the event. In this experience, I think that for them, everything was virtual : the machines, the network, the support, the availability, the visitors and finally the profit. I really can't say what you could play on to improve quality. After having spent 3 full nights working with them on their machine, no sensible trend appeared whatever we did. I think the real knobs are outside your scope, on the other side of the VM :-/ Regards, Willy -- Name: Joseph A. Williams Email: j...@joetify.com Blog: http://www.joeandmotorboat.com/
Re: Tuning HAProxy on EC2 instances?
On 1/31/10 8:01 PM, Alexander Staubo wrote: We might end up deciding to use a dedicated, non-virtual hosting provider. That assumes we can find one that lets us cheaply and quickly (eg., within a day or two) add or remove new machines. There are a bunch of providers like that in the US, but I don't know of any reputable ones in Europe. Do you know of any? Alexander, Another option is Voxel.net - they can deploy across nodes across the United States, in Amsterdam, and Singapore. They also have a detailed API (XML, JSON), Apache modules (mod_cdn), and pricing is more expensive than Amazon but still pretty good. -- Harvey
Re: Tuning HAProxy on EC2 instances?
Dnia 2010-02-01, pon o godzinie 02:01 +0100, Alexander Staubo pisze: > On Mon, Feb 1, 2010 at 12:14 AM, Willy Tarreau wrote: > > well, last year I helped some guys in charge of a world wide sports > > event which was hosted there. The performance was terrible. Completely > > unstable. [snip] > > > > In this experience, I think that for them, everything was virtual : > > the machines, the network, the support, the availability, the visitors > > and finally the profit. > > Ouch. That's something of a horror story. Thanks for the summary. > This, and several recent blog posts about EC2 performance issues, is > making me want to reconsider EC2. > > The negatives have not been reflected in our own testing of EC2, but > then so far we have only dealt with single, standalone instances which > only depend on external traffic. There is a lot of evidence that EC2 > suffers from internal network latency as well as being overcrowded, at > least in the US. We will need to run some comprehensive performance > tests with multiple instances. > > But it's hard to ignore the myriad of services that Amazon provides. > S3, EBS, autoscaling, Elastic IPs, geographic CDN -- those are all > things we want. A virtually infinite supply of storage space through > EBS is a particularly attractive proposition, and one which I think > very few dedicated hosting companies can provide. We don't want to pay > through the nose for some kind of half-assed SAN setup. > > We might end up deciding to use a dedicated, non-virtual hosting > provider. That assumes we can find one that lets us cheaply and > quickly (eg., within a day or two) add or remove new machines. There > are a bunch of providers like that in the US, but I don't know of any > reputable ones in Europe. Do you know of any? > OVH is quite okay. From VPS i'd recommend linode.com (they have data centers in europe and US), much faster than Amazon, and also have some kind of API -- Mariusz Gronczewski (XANi) GnuPG: 0xEA8ACE64 http://devrandom.pl signature.asc Description: To jest część wiadomości podpisana cyfrowo
Re: Tuning HAProxy on EC2 instances?
On Mon, Feb 1, 2010 at 12:14 AM, Willy Tarreau wrote: > well, last year I helped some guys in charge of a world wide sports > event which was hosted there. The performance was terrible. Completely > unstable. [snip] > > In this experience, I think that for them, everything was virtual : > the machines, the network, the support, the availability, the visitors > and finally the profit. Ouch. That's something of a horror story. Thanks for the summary. This, and several recent blog posts about EC2 performance issues, is making me want to reconsider EC2. The negatives have not been reflected in our own testing of EC2, but then so far we have only dealt with single, standalone instances which only depend on external traffic. There is a lot of evidence that EC2 suffers from internal network latency as well as being overcrowded, at least in the US. We will need to run some comprehensive performance tests with multiple instances. But it's hard to ignore the myriad of services that Amazon provides. S3, EBS, autoscaling, Elastic IPs, geographic CDN -- those are all things we want. A virtually infinite supply of storage space through EBS is a particularly attractive proposition, and one which I think very few dedicated hosting companies can provide. We don't want to pay through the nose for some kind of half-assed SAN setup. We might end up deciding to use a dedicated, non-virtual hosting provider. That assumes we can find one that lets us cheaply and quickly (eg., within a day or two) add or remove new machines. There are a bunch of providers like that in the US, but I don't know of any reputable ones in Europe. Do you know of any?
Re: Tuning HAProxy on EC2 instances?
Hi Alexander, On Sun, Jan 31, 2010 at 11:36:02PM +0100, Alexander Staubo wrote: > Has anyone any experience tuning HAProxy for performance when running > on Amazon EC2 instances? For example, are there any kernel parameters > that should be tuned differently, or are some instance types better > than others? Does HAProxy generally perform well on EC2? well, last year I helped some guys in charge of a world wide sports event which was hosted there. The performance was terrible. Completely unstable. It was impossible to tune anything. Ping times would vary a lot. It was impossible to know where the bottlenecks were, because every machine was showing limited performance in turn without necessarily having its CPU saturated. It was noticed that the internal network was at least faulty, because the observed network congestions were not constat and moving between machines. Sometimes it was even almost impossible to type in SSH. We also discovered that when they bought new nodes, some of them were under massive attacks, most likely because people who are attacked quickly drop the nodes with the IPs that belong to them and create new ones. So the attacked ones will be picked by the next customer... Finally they moved to a real hosting company with real machines and real performance in order to be able to participate at least to a little part of the event. In this experience, I think that for them, everything was virtual : the machines, the network, the support, the availability, the visitors and finally the profit. I really can't say what you could play on to improve quality. After having spent 3 full nights working with them on their machine, no sensible trend appeared whatever we did. I think the real knobs are outside your scope, on the other side of the VM :-/ Regards, Willy
Tuning HAProxy on EC2 instances?
Has anyone any experience tuning HAProxy for performance when running on Amazon EC2 instances? For example, are there any kernel parameters that should be tuned differently, or are some instance types better than others? Does HAProxy generally perform well on EC2?
