Slight shift in our application strategy
Background: Shortly after we started scaling out to remote sites we noticed some parts of some of our applications had issues over the vpn link. Initial ticket creation was a bit off as to what we were looking at but: https://fedorahosted.org/fedora-infrastructure/ticket/281 To try to keep this short: after lots of tests we discovered that applications that run lots of queries are the core of our issue. We have ok bandwidth to all of our sites, but latency is high enough that it's become too expensive to actively run applications at these sites. Every query that gets run at a remote site seems to take a minimum of .3 to .5 seconds for the complete round-trip. As we mature and as features get in lots of our apps need more queries. We can and should go through and make these more efficient but that's going to happen over a long time. We just don't have the number of people we need to do trends on each page of each application and convert all the sql to its most efficient. Instead we're going to convert all of our remote application servers to passive/backup servers. Up until now we've generally been using our remote sites to scale load. Now though we can't really do that. They're an important role for being fairly HA (our SPOF is still our data layer). Having a multi-master data layer of postgres and mysql just won't be a win for us at our size at this time. So what does this mean for the future? Our scaling issues at our app layer will just have to be in a centralized location. This won't scale forever but I think for the near and middle term in Fedora's future it's what we're going to have to bank on. We can continue to focus on better caching at our proxy layer which will continue to be active at each remote site. All of what I've written here is probably very obvious to most of you, and it is. The difference now is we have some much better data concerning the interaction between our app servers and the data layer and better metrics for how long those interactions take. So darnit, I'm not going to spend all the time I did with testing and metrics and not write some long summary about it! :) -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: opengroupware evaluation
On Thu, 19 Feb 2009, susmit shannigrahi wrote: > On Thu, Feb 19, 2009 at 10:33 PM, Mike McGrath wrote: > > I'd like one of our volunteers to install opengroupware on a publictest > > server for our evaluation. > > > > https://fedorahosted.org/fedora-infrastructure/ticket/1197 > > > > This is going to take between 5 and 15 hours a week. Don't volunteer > > unless you can commit that much time to it. > > I can commit some time from the coming week. (23rd.) > If that is ok, I can do it. > It has been long since I have something for Infra. :) > Thanks. > Just checking in, how did things go? Any updates? Any one else interested in joining susmit to get this up and running? -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: opengroupware evaluation
I can spare a few hours a week to work on it. I might need someone to mentor me on it if that is an option because I am new other than the time that I was looking at the redundant OpenVPN scenarios with you. What groups would I need to apply for? Username is tjdavisbz. TJ Davis On Fri, Feb 27, 2009 at 4:52 PM, Mike McGrath wrote: > > On Thu, 19 Feb 2009, susmit shannigrahi wrote: > > > On Thu, Feb 19, 2009 at 10:33 PM, Mike McGrath > wrote: > > > I'd like one of our volunteers to install opengroupware on a publictest > > > server for our evaluation. > > > > > > https://fedorahosted.org/fedora-infrastructure/ticket/1197 > > > > > > This is going to take between 5 and 15 hours a week. Don't volunteer > > > unless you can commit that much time to it. > > > > I can commit some time from the coming week. (23rd.) > > If that is ok, I can do it. > > It has been long since I have something for Infra. :) > > Thanks. > > > > Just checking in, how did things go? Any updates? > > Any one else interested in joining susmit to get this up and running? > >-Mike > > ___ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: opengroupware evaluation
2009/2/27 TJ Davis : > I can spare a few hours a week to work on it. I might need someone to > mentor me on it if that is an option because I am new other than the time > that I was looking at the redundant OpenVPN scenarios with you. What groups > would I need to apply for? Username is tjdavisbz. > > > TJ Davis > > TJ, Welcome. Join the sysadmin-test group in FAS. I'm in the same boat as you, a few hours to spare every week. Cheers, Clint ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: opengroupware evaluation
Thanks, I have applied for sysadmin-test. TJ Davis On Fri, Feb 27, 2009 at 6:44 PM, Clint Savage wrote: > 2009/2/27 TJ Davis : > > I can spare a few hours a week to work on it. I might need someone to > > mentor me on it if that is an option because I am new other than the time > > that I was looking at the redundant OpenVPN scenarios with you. What > groups > > would I need to apply for? Username is tjdavisbz. > > > > > > TJ Davis > > > > > TJ, > > Welcome. Join the sysadmin-test group in FAS. I'm in the same boat > as you, a few hours to spare every week. > > Cheers, > > Clint > > ___ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: opengroupware evaluation
On Fri, 27 Feb 2009, TJ Davis wrote: > Thanks, I have applied for sysadmin-test. > Right now susmit has lead on this but I'm not sure what his time constraints are over the next couple of weeks. If he's busy I'll just make sure you guys all have access and can hammer away at it a couple of hours at a time. -Mike > > TJ Davis > > > > On Fri, Feb 27, 2009 at 6:44 PM, Clint Savage wrote: > 2009/2/27 TJ Davis : > > I can spare a few hours a week to work on it. I might need someone to > > mentor me on it if that is an option because I am new other than the > time > > that I was looking at the redundant OpenVPN scenarios with you. What > groups > > would I need to apply for? Username is tjdavisbz. > > > > > > TJ Davis > > > > > TJ, > > Welcome. Join the sysadmin-test group in FAS. I'm in the same boat > as you, a few hours to spare every week. > > Cheers, > > Clint > > ___ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > >___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
