Thanks Stanton -- more comments inline. >-----Original Message----- >From: Stanton Sievers [mailto:ssiev...@us.ibm.com] >Sent: Thursday, April 26, 2012 1:19 PM >To: dev@shindig.apache.org >Subject: Re: Shindig HttpFetcher > >Hi Jesse, > >I'm looking at BasicHttpFetcher and it seems like someone had this idea >(or something similar) in mind when implementing it. There's logic in the >fetch method to check for slow responses and call a protected >slowResponseWarning method that currently only logs a warning if a >response took too long. The comments on slowResponseWarning seem to >suggest overriding it "if you want to do something other than logging a >warning". From an implementation standpoint, you could keep track of slow >responses via this method and then override the fetch method to check if >the incoming request has been flagged as slow before calling super.fetch. > >This still doesn't answer your question about how to best figure out what >apps are slow when using a reverse proxy,
Right -- that's the piece that I'm not sure how best to handle... I appreciate your response though! > but I think that's an issue that >doesn't need to be solved in Shindig. It's pretty much implementation >specific at that point. I was thinking if we could come up with a generic enough solution it might be useful for many/most implementers. If we come up with something generic enough I'll put a review out and we can decide if it's useful/generic enough for shindig core. >I don't think I really answered your question, but figured I would throw >in my 2 cents anyways. :) > >Thanks, >-Stanton > > > >From: "Ciancetta, Jesse E." <jc...@mitre.org> >To: "dev@shindig.apache.org" <dev@shindig.apache.org>, >Date: 04/25/2012 07:44 >Subject: Shindig HttpFetcher > > > >Hi All, > >We're running into an issue with our HttpFetcher implementation where slow >responses to outgoing HTTP requests are crashing our shindig server. Every >once in a great while one of the services our gadgets call on for data >will have some kind of issue and start returning really slowly which >causes a backup of requests in the fetcher to the point that shindig >starts to become unresponsive. > >So we're basically looking to implement some kind of dynamic throttling in >our fetcher where we can detect that some service is having a problem and >throttle down requests to that service until its response times return >back to normal. However, one detail that makes our case a little trickier >that normal (or maybe not depending on how common this practice is) is >that we mask many different services behind a single host using a reverse >proxy -- so all of these URL's on the same host may actually be hitting >different servers on the backend: > >host.example.com/some-app >host.example.com/some-collection-of-similar-apps/app-1/api/foo >host.example.com/some-collection-of-similar-apps/app-1/api/foo/bar >host.example.com/some-collection-of-similar-apps/app-1/api/foo/... >host.example.com/some-collection-of-similar-apps/app-2 >host.example.com/some-collection-of-similar-apps/app-2/foo >host.example.com/different-app >... > >That makes truly dynamic throttling kind of tricky since the paths can >vary arbitrarily -- so for example if this app was the one that was slow: > >host.example.com/some-collection-of-similar-apps/app-2 > >we want to throttle down requests to just that one app but let the others >continue through. > >One other detail -- some of the apps behind our reverse proxy are legacy >apps which are obnoxiously slow so we have to set the timeout on our >fetcher to an obnoxiously high value to accommodate those legacy services >which makes this problem just that much worse... > >The only solution I've come up with so far is to define some sort of >regular expressions or something that we can use to figure out what app a >given URL belongs to and then using that we can see that app-2 is slow and >start throttling requests to it -- but that's a very manual/specific >solution to the problem and we were hoping to come up with something >more >generic that we could push back into shindig core. > >Has anyone else has faced similar issues and/or have any suggestions on >how to go about handling this? > >Thanks! > >--Jesse >
