On Aug 15, 2012, at 6:53 PM, Rob Weir wrote: > On Wed, Aug 15, 2012 at 7:31 PM, Kay Schenk <kay.sch...@gmail.com> wrote: >> On Wed, Aug 15, 2012 at 4:04 PM, Rob Weir <robw...@apache.org> wrote: >> >>> On Wed, Aug 15, 2012 at 5:29 PM, Kay Schenk <kay.sch...@gmail.com> wrote: >>>> On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <robw...@apache.org> wrote: >>>> >>>>> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <dave2w...@comcast.net> >>>>> wrote: >>>>>> >>>>>> On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote: >>>>>> >>>>>>> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400: >>>>>>>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf < >>> d...@daniel.shahaf.name> >>>>> wrote: >>>>>>>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 >>> +0200: >>>>>>>>>> Is it possible that somebody from the Apache Infrastructure can >>>>>>>>>> provide a view on which URL the traffic load was soo high that the >>>>>>>>>> servers got in trouble? >>>>>>>>>> >>>>>>>>> >>>>>>>>> POST requests to /ProductUpdateService/check.Update file >>>>>>>>> >>>>>>>> >>>>>>>> For which subdomain, which UpdateXX.openoffice.org ? >>>>>>> >>>>>>> The access log doesn't say, and the error log has >>>>>>> >>>>>>> % fgrep /ProductUpdateService/check.Update error_log | sed -e >>>>> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c >>>>>>> >>>>>>> EU: >>>>>>> 232046 update30 >>>>>>> 35548 update34 >>>>>>> 76543 update35 >>>>>>> >>>>>>> >>>>>>> US: >>>>>>> 198996 update30 >>>>>>> 33450 update34 >>>>>>> 71117 update35 >>>>>>> 0 update36 >>>>>> >>>>>> We don't see update32 because those do not get redirected in the same >>>>> way because there is no ooo-site/trunk/content/projects/update32 >>>>> >>>> >>>> uh oh...this should have been setup before and Oliver said he requested >>>> this in the first post here. >>>> >>>> And you're now saying that all the previous ones have been reverted? >>>> >>>> I think we were OK until this last one, right? update32? >>>> >>>> I think the others should be re-established as they weren't causing >>>> problems, were they? >>>> >>>> The thing is unless we go back to the code for OOo 3.1, we don't know >>> what >>>> it's doing. >>>> >>>> When I asked about this when we had issues for OOo 3.0, I was told it was >>>> fixed in OOo 3.2, so maybe 3.1. has the same issues? >>>> >>>> >>>> >>>>> >>>>>> ./update/aoo341/check.Update >>>>>> ./update/ProductUpdateService/check.Update >>>>>> ./update30/ProductUpdateService/check.Update >>>>>> ./update34/ProductUpdateService/check.Update >>>>>> ./update34/ProductUpdateService/test.Update >>>>>> ./update35/ProductUpdateService/check.Update >>>>>> ./update35/ProductUpdateService/test.Update >>>>>> ./update36/ProductUpdateService/check.Update >>>>>> ./update38/ProductUpdateService/check.Update >>>>>> >>>>>> It looks like 34 and 35 have been trouble, but not as bad as 30. >>>>>> >>>>> >>>>> But haven't 34 and 35 been in production since early July? We've >>>>> certainly seen plenty of downloads triggered by them. They should not >>>>> be giving any errors, since the requests resolve to files on our site. >>>>> >>>>> I wonder, could the errors in those be caused by the outage caused by >>>>> the errors in update30? >>>>> >>>> >>>> Rob...update 30 is completely out of the question, and we simply can not >>>> do it, and reverted it within hours when I first requested it. >>>> >>> >>> What is the issue with update 30? The fact that it does a POST? I >>> don't that would rule it out altogether. But we would need to treat >>> it specially. For example, we could redirect to an isolated server, >>> at Apache or outside, that is able/willing to handle it. If we run it >>> for a month or two we should get the bulk of the upgrades. >>> >>> Or was there some other issue? >>> >> >> The Apache web server, of which AOO is a part, does not allow POSTs so when >> I had infra enable this and redirect the old update30 to the web server, it >> caused MANY errors in a very short period of time (about an hour) and Joe >> reverted it rather quickly. THis was like back in early March or >> something when I was playing around. The update feed itself didn't even DO >> anything but redirect them to a URL (in theory) it was the POST in the code >> for OOo 3.0 that caused all the havoc. When I inquired about this on this >> list, I was told yes, this WAS the case for 3.0 but had been fixed with, I >> thought 3.2. >> >> Anyway, as far as I know, this is the only issue. >> >> I was pretty wary initially about running the feed through the web server >> but was told we should be fine (this after Joe reverted the update30 in >> March) -- and I think we have been for the most part. But, yes, we need >> another box with a web server that would accept POSTs to deal with this -- >> 3.0, and it looks like 3.1. >> > > OK. That's the impression I had as well -- POST was disabled in > general on ASF servers. We're not going to be able to change that. > But the traffic from permitted POSTs will be far less than the errors > and retries generated from disallowed POSTs. So if we can find a > volunteer to host this traffic on their website, then we could have > the redirects go there rather than to openoffice.org or staroffice.de. > This would not need to be a host volunteering to do this forever. I > think if we ran it for 2 months we'd get almost everyone, since the > upgrade checks occur once a week by default. > > But we can discuss this more after AOO 3.4.1 is out. No sense > tempting the OOo 3.1 users to upgrade to AOO 3.4.0 today and then send > them a notification AOO 3.4.1 a week later. That would be annoying.
Agreed. It may be we can get an Infra VM - ooo-slop. Put a simple apache server with POST and keep the logs for our own analysis. If it crashes only the update process is harmed. I will close the existing JIRA and then open another explicitly for when we have this decided. Regards, Dave > > -Rob > >> >>>> There IS an update30 directory there but it isn't actually being used, >>> and >>>> is just a dummy file anyway. Maybe we should just delete this one so we >>>> won't get confused about this one anymore. It was setup in early stages >>> of >>>> testing. >>>> >>>> Should I just delete ./update30/ProductUpdateService/check.Update -- I >>> mean >>>> the whole directory. >>>> >>>> >>>>> -Rob >>>>> >>>>>> Regards, >>>>>> Dave >>>>> >>>> >>>> >>>> >>>> -- >>>> >>> ---------------------------------------------------------------------------------------- >>>> MzK >>>> >>>> "Never express yourself more clearly than you are able to think." >>>> >>> -- >>>> Niels Bohr >>> >> >> >> >> -- >> ---------------------------------------------------------------------------------------- >> MzK >> >> "Never express yourself more clearly than you are able to think." >> -- >> Niels Bohr
