Re: [lopsa-tech] Cloud Databases?
On Thu, Oct 7, 2010 at 2:52 PM, Atom Powers wrote: > One of the COs in my company keeps asking "why don't we put it all in > the cloud"? I'm running out of good answers. > > Situation: > We have four locations (US-West, Singapore, Korea, Spain) and two > datacenters (US-West, Singapore). We have a database-driven > application (Content Management System) that is mission-critical to > all locations and all potential customers all over the world. > Currently that entire system (MySQL database and Apache web servers) > are all located in the US; performance in Asia and Spain is dismal. > > My boss, having no idea what "the cloud" actually is, wants to move > the entire system into "the cloud". After some investigation, we have > determine that some aspects of the system can be hosted on Amazon AWS > or similar. But we have big doubts about database performance, big > doubts. > > Question: > What are your experiences with cloud-based (or cloud-capable) content > management systems? > What options are there for putting a database-driven application in "the > cloud"? > Having recently done (part of) this analysis, I have some opinions. First, your main problem is that the manager doesn't know what "cloud" is. "Cloud" as you are discussing is infrastructure as as service (IaaS), as opposed to Software as a Service (SaaS), etc. IaaS is usually no more than outsourcing hardware resources. You are still responsible for *everything*, and yet you have less control. You have no control or visibility into the network, for example. A reasonably accessible overview of "cloud" and its flavors is in this guide from the Cloud Security Alliance, especially the first 20 pages or so. http://www.cloudsecurityalliance.org/csaguide.pdf Second, your problem is really performance of your application over high-latency links. Specifically, your US-based app is slow as perceived from Span and Asia. Moving to the cloud won't solve this problem. The servers still have to live somewhere. Granted, you could move the application to some other continent, but that doesn't require cloud. You could go into managed hosting (or the "cloud") in Europe or Asia, but then performance would likely still be bad for the US and whichever region didn't have the app locally. You will need to address the app's network performance no matter how or where you host it. Cloud isn't going to solve this. You might need to do more caching, or optimize page loading, for example. Re-writing the app to make a single database call (stored procedures perhaps) per page as opposed to making dozens of separate calls (from the client) per page will give you an amazing performance boost. Caching may help as well, but definitely look at how many DB calls it takes to fill a page, especially if this is an Ajax-y app where the DB calls are made from the client or an intermediate web sever. For example, if your web server front end was located in Spain to be near the client, but is 113ms from the DB server in the US, then the page load time will be the number of DB calls from the web server * the network latency. For 20 DB calls per page, this is over 2.25 seconds to load the page. (I see 76ms from east coast to .EU and 151ms from west coast to EU, average is 113ms.) Third, database applications can be I/O ntensive. An attached EBS volume (for AWS EC2) will likely perform more like a faster IDE drive than a RAID array. See ( http://af-design.com/blog/2009/02/27/amazon-ec2-disk-performance ). Don't forget that you also pay for each I/O that you read/write to the EBS. It's a $0.10 per million I/Os, but that might add up over time. If your app already wants high-speed SAN, or has a big fast RAID, then benchmark carefully. Fourth, depending on the size of the instance, it may be more or less expensive than your own hardware. Cloud vendors always show the price of their smallest instances but those are completely unsuitable for a database of any size. Look at this page for the pricing model: http://aws.amazon.com/ec2/pricing/ Now, what are your costs. Let's assume that you are running your own hardware, in your own datacenter, using UNIX. Typical DB server for us is 4 cores, 12 GB RAM and a few drives. HW RAID usually. Here's a possible config: http://h10010.www1.hp.com/wwpc/us/en/sm/WF25a/15351-15351-3328412-241644-241475-4091408.html#model-diff It's $2200 plus drives, so call it $3.5K max for the server. Depreciate that over 36 months and you get $97.22/month. That's $0.144/hour (4 week, 28 day month). Now look at the AWS pricing... There is no usable instance that beats that price. As long as your other operational costs aren't too high, you are cheaper That said, you will get quite a few operational features from running MySQL in the cloud, such as quick snapshots, ability to move your DB servers, etc. Short answer: Going to cloud is very unlikely to solve your problem :-) ___ Tech mailing
Re: [lopsa-tech] Cloud Databases?
On Thu, Oct 7, 2010 at 6:26 PM, Atom Powers wrote: >> You also mentioned file access time - could you describe that problem a bit >> more? > > Files are primarily served though either CIFS (samba) or HTTP (Web > CMS) and files can be very large, up to 1GB. Rough estimates show that > a 200ms latency increases the download time by up to 20%, > proportionally larger for smaller files. "CIFS" makes me think many of these things are internal facing apps? Have you looked at putting WAN accelerators on the links in-between sites? At $lastjob we used Steelheads to turn cruddy t-1's in the middle of nowhere into respectable links and for ds3's between major sites they turned CIFS traffic into nearly-next-door. Short of that, tuning/fixing MTU's, pMTUd, and VPN settings to cope with latency / be more optimal and dealing with bandwidth hogs always tend to help. For me, cloudifying existing entrenched enterprise level apps isn't my idea of fun. So, I'd ensure that the entire network path between $far and $near is as optimal as possible before doing major surgery to the application stack. After that I'd start looking for slack in the edges of the apps that can be solved with CDN-like or proxy like infrastructure at sites to reduce calls across the long links. It may be that the 20k of html gets to the client very quickly, but the 75k of images in 4k chunks take forever to load. (Note, some WAN accelerators will do this sort of thing for you too). Another way to solve this kind of issue for people in really, really far flung places is with Citrix/RDP/VNC/NoMachine to let the user remote in to a machine closer to the data. These kinds of services stream and compress pretty well and for the office of users 6k miles away who need to importantly run some critical reports once a reporting period, they can be a perfect fit. -n -- --- nathan hruby metaphysically wrinkle-free --- ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
Re: [lopsa-tech] Cloud Databases?
Ensure your application is designed to be highly redundant as with Amazon and the "cloud" in general, you will lose an instance at any time and it is very possible it will not come back (And sometimes you'll even have trouble killing it if it does go down). IO performance with EBS will also vary wildly. It's recommended to stripe across EBS volumes with mdraid to get better performance. We have one LAMP site with AWS now, it's worked well for the most part, but we do run into issues atleast once a month related to either network issues or losing an instance or two. We haven't had any performance issues as the site itself is not too resource intensive. Unfortunately we don't have a support contract so that makes things a bit harder to deal with when issues arise. There have been a few times where an instance has just disappeared and I've tried restarting it a few times and then gone through to start launching a new one when the other would magically come back online an hour or two later. Depending on how well of a shop you run, bringing up new instances may be as simple as having your config mgmt system do all the work and its no big deal to have instances drop off. - Justin Lintz On Thu, Oct 7, 2010 at 5:52 PM, Atom Powers wrote: > One of the COs in my company keeps asking "why don't we put it all in > the cloud"? I'm running out of good answers. > > Situation: > We have four locations (US-West, Singapore, Korea, Spain) and two > datacenters (US-West, Singapore). We have a database-driven > application (Content Management System) that is mission-critical to > all locations and all potential customers all over the world. > Currently that entire system (MySQL database and Apache web servers) > are all located in the US; performance in Asia and Spain is dismal. > > My boss, having no idea what "the cloud" actually is, wants to move > the entire system into "the cloud". After some investigation, we have > determine that some aspects of the system can be hosted on Amazon AWS > or similar. But we have big doubts about database performance, big > doubts. > > Question: > What are your experiences with cloud-based (or cloud-capable) content > management systems? > What options are there for putting a database-driven application in "the > cloud"? > > -- > Perfection is just a word I use occasionally with mustard. > --Atom Powers-- > ___ > Tech mailing list > Tech@lopsa.org > http://lopsa.org/cgi-bin/mailman/listinfo/tech > This list provided by the League of Professional System Administrators > http://lopsa.org/ > ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
Re: [lopsa-tech] Cloud Databases?
On Thu, Oct 7, 2010 at 4:12 PM, Matthew Barr wrote: > Ok, a few basic questions. Basic Answers... > What's the workload? What portion of the db access is read vs write? What > portions of write traffic is US west vs other locations? Currently 90% US but we expect it to become 50% US, 30% SG, 20% Spain. The majority of it is read, but we also have collaboration projects that have a lot of writes. > You also mentioned file access time - could you describe that problem a bit > more? Files are primarily served though either CIFS (samba) or HTTP (Web CMS) and files can be very large, up to 1GB. Rough estimates show that a 200ms latency increases the download time by up to 20%, proportionally larger for smaller files. > Basically, if you've got a reasonably normal CMS, much of the traffic is > read. You've already got asiapac datacenter space, so some replication > techniques may be able to get big wins. Mysql slaves, prepositioning files > via sync or CDN, some global load balancing tricks... This is the direction that we have been trying to move in. But lately funding and manpower have been an issue. I believe my real challenge is educating the COs about risk and cost, but I also need to know if what they are asking for is even remotely possible. > Cloud won't help- you've already got 2 sites, and aren't utilizing them. And two more sites without datacenters. I can put slaves and caching server is Singapore, which will help Korea, but I can't do anything about Spain. -- Perfection is just a word I use occasionally with mustard. --Atom Powers-- ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
Re: [lopsa-tech] Cloud Databases?
Ok, a few basic questions. What's the workload? What portion of the db access is read vs write? What portions of write traffic is US west vs other locations? You also mentioned file access time - could you describe that problem a bit more? Basically, if you've got a reasonably normal CMS, much of the traffic is read. You've already got asiapac datacenter space, so some replication techniques may be able to get big wins. Mysql slaves, prepositioning files via sync or CDN, some global load balancing tricks... Cloud won't help- you've already got 2 sites, and aren't utilizing them. You'll need to look at these techs with the cloud, you might as well start there... Matthew Sent from my iPad On Oct 7, 2010, at 5:52 PM, Atom Powers wrote: > One of the COs in my company keeps asking "why don't we put it all in > the cloud"? I'm running out of good answers. > > Situation: > We have four locations (US-West, Singapore, Korea, Spain) and two > datacenters (US-West, Singapore). We have a database-driven > application (Content Management System) that is mission-critical to > all locations and all potential customers all over the world. > Currently that entire system (MySQL database and Apache web servers) > are all located in the US; performance in Asia and Spain is dismal. > > My boss, having no idea what "the cloud" actually is, wants to move > the entire system into "the cloud". After some investigation, we have > determine that some aspects of the system can be hosted on Amazon AWS > or similar. But we have big doubts about database performance, big > doubts. > > Question: > What are your experiences with cloud-based (or cloud-capable) content > management systems? > What options are there for putting a database-driven application in "the > cloud"? > > -- > Perfection is just a word I use occasionally with mustard. > --Atom Powers-- > ___ > Tech mailing list > Tech@lopsa.org > http://lopsa.org/cgi-bin/mailman/listinfo/tech > This list provided by the League of Professional System Administrators > http://lopsa.org/ ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
Re: [lopsa-tech] Cloud Databases?
I should say that our performance bottleneck is latency. 200ms to Singapore means that even simple file transfers (with bad protocols like CIFS and HTTP) take longer than expected. (And isn't expectation the key measure of performance with COs?) I'm not really clear on how a "cloud" service would overcome that bottleneck. Or how that would be any better than keeping a "mirror" of the database and application in each of our datacenters. (I imagine keeping a database synchronized with that kind of latency would be painful, if it was my datacenter or Amazon's or a VPS.) On Thu, Oct 7, 2010 at 2:52 PM, Atom Powers wrote: > One of the COs in my company keeps asking "why don't we put it all in > the cloud"? I'm running out of good answers. > > Situation: > We have four locations (US-West, Singapore, Korea, Spain) and two > datacenters (US-West, Singapore). We have a database-driven > application (Content Management System) that is mission-critical to > all locations and all potential customers all over the world. > Currently that entire system (MySQL database and Apache web servers) > are all located in the US; performance in Asia and Spain is dismal. > > My boss, having no idea what "the cloud" actually is, wants to move > the entire system into "the cloud". After some investigation, we have > determine that some aspects of the system can be hosted on Amazon AWS > or similar. But we have big doubts about database performance, big > doubts. > > Question: > What are your experiences with cloud-based (or cloud-capable) content > management systems? > What options are there for putting a database-driven application in "the > cloud"? > > -- > Perfection is just a word I use occasionally with mustard. > --Atom Powers-- > -- Perfection is just a word I use occasionally with mustard. --Atom Powers-- ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
Re: [lopsa-tech] Cloud Databases?
Atom Powers writes: > One of the COs in my company keeps asking "why don't we put it all in > the cloud"? I'm running out of good answers. Cost it out. Depending on how much dealing with hardware costs you, there are cases where the cloud can save you money; but usually, if you plan on leaving a server up for more than four months, it is cheaper to buy a server and co-locate it, before you count the inconvenience of dealing with physical hardware. The other misconception people have is that ec2 will eliminate your SysAdmin. It won't. It will reduce the load on the hardware guy, but you still need people who understand *NIX. > Question: > What are your experiences with cloud-based (or cloud-capable) content > management systems? > What options are there for putting a database-driven application in "the > cloud"? I know a lot of people who buy and co-locate beefy servers for database servers, who then go and use AWS or similar for web front ends. Most people I know doing databases in 'the cloud' use the "database 2.0" stuff. Couchdb and the like. NoSQL type stuff. I have a bunch of customers running databases on my VPS platform and I can tell you that sharing disk gives you suboptimal performance. the thing is, if you have two people splitting a SATA disk down the middle, sure, each guy gets half the space, but each person gets a whole lot less than half the performance. SATA disk is pretty good for sequential transfer, and operating systems go to great lengths to attempt to turn random writes into sequential writes. The problem is that if you've got two different virtuals each streaming sequential writes to the same disk, you get what is basically random access to those disks, as it's got to switch fairly quickly from writing one stream to writing the other. I am given to understand that Linode deals with this by keeping count of your IOPS, and if you trip a threshold, they warn you. If you trip the next threshold, they start actively limiting your disk I/O. This, I think, is a good idea, and something I will likely implement myself. but the point is that sharing spinning disk is not good for I/O performance. (now, how well does amazon EBS stack up to a database write load vs. local disk? I don't know. I'd be interested to hear, though. Amazon also offers SAS disks, I think, and sas performance degrades less when it becomes random (vs sata) but a shared sas disk is still never going to beat a unshared sas disk for performance.) Now, this isn't to say that the cloud is always a bad idea; if you have compute needs that vary by the hour, it's goddamn difficult to beat amazon.com. And if you need a dev box with 512MiB ram, there is no way you are going to be able to host even a free 1u for what it costs to get a VPS of that size. ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
Re: [lopsa-tech] Cloud Databases?
On 10/7/2010 2:52 PM, Atom Powers wrote: > One of the COs in my company keeps asking "why don't we put it all in > the cloud"? I'm running out of good answers. Lucky you. It's raining outside, and although there are things I could do, this is more interesting. I'll give you some brief answers now, and may return later. > ...We have a database-driven > application (Content Management System) that is mission-critical to > all locations and all potential customers all over the world. Oh, you used some of my favorite words. Mission critical database systems should not be left to the vagaries of the cloud. I'm sure that you've already seen this page. Right now, it's all quite beautiful, but I've seen some unpleasant moments over time. It's a pity that the chart only covers back for 35 days, since there haven't been any real fun events for the last few weeks. One of my very favorite toys is http://deals.woot.com/ which lives entirely in the cloud. It's dot net stuffs built on top of the cloud, so it has automatic extra crazy built in, and it simply never ceases to entertain me. My favorite moment was when UltraDNS failures just brought them to their knees (I posted in the forums because Deals was toast). http://www.woot.com/forums/viewpost.aspx?PostID=3686384 For a while I was even following some of the developers on Twitter, but that was ages ago. You could always just create your *own* cloud. -- The difference between a Miracle and a Fact is exactly the difference between a mermaid and a seal. Mark Twain ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
[lopsa-tech] Cloud Databases?
One of the COs in my company keeps asking "why don't we put it all in the cloud"? I'm running out of good answers. Situation: We have four locations (US-West, Singapore, Korea, Spain) and two datacenters (US-West, Singapore). We have a database-driven application (Content Management System) that is mission-critical to all locations and all potential customers all over the world. Currently that entire system (MySQL database and Apache web servers) are all located in the US; performance in Asia and Spain is dismal. My boss, having no idea what "the cloud" actually is, wants to move the entire system into "the cloud". After some investigation, we have determine that some aspects of the system can be hosted on Amazon AWS or similar. But we have big doubts about database performance, big doubts. Question: What are your experiences with cloud-based (or cloud-capable) content management systems? What options are there for putting a database-driven application in "the cloud"? -- Perfection is just a word I use occasionally with mustard. --Atom Powers-- ___ Tech mailing list Tech@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
