Okay, I have a couple ideas now where the problem lies. I want to give you a heads up that we'll be releasing beta 1 soon, and it won't include a fix for this. This is going to be a much larger problem to tackle and I don't want to hold up this release (since it has many very important fixes in it). I hope to get to this one for beta 2.
Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.review-board.org VMware, Inc. - http://www.vmware.com On Fri, Mar 27, 2009 at 10:27 AM, ciaomary <ciaom...@gmail.com> wrote: > > We typically only see perf problems on reviews with large diffs and > large number of reviews. One that has given us a lot of trouble has a > downloadable diff of 1.6MB, 138 different files (with an average file > size of ~30K), and hundreds+ review comments. (Bringing up diffs for > deleted files give the biggest load and slow down.) > > Under Admin Settings I've changed the "Paginate by" value from 20 down > to just 3 which has helped a lot. It is set to the default 5 "Lines of > Context". Also increasing the RAM on our server from 4G to 16G has > helped - RB takes a lot of server memory under this kind of load! > After all this we're still seeing these big reviews taking 45-60 > seconds to load, pegging local cpu during that time. > > I suspect you could reproduce this by setting the "Paginate by" value > to something very high. (Try 55 for your 55 page diff.) I believe the > problem is only on reviews with lots of comments as well. Our use case > has been that if there is a really large diff, there are always a lot > of review comments as well. > > Thanks! > > On Mar 27, 1:38 am, Christian Hammond <chip...@chipx86.com> wrote: > > So I just looked at a diff on our server that spans 55 pages, and as far > as > > loading diffs goes, it's not at all slow. How many pages did you say > yours > > were on average? > > > > When looking at diffs, do you have it set to expand the entire file, or > show > > the collapsed version? > > > > Are you seeing slowdown on large diffs that don't have any reviews, or is > it > > just when there's a lot of reviews for that revision of the diff? > > > > Christian > > > > -- > > Christian Hammond - chip...@chipx86.com > > Review Board -http://www.review-board.org > > VMware, Inc. -http://www.vmware.com > > > > On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond <chip...@chipx86.com > >wrote: > > > > > > > > > I'm still looking into it. So far I have not seen the slowdown you have > > > described, but will continue to try to figure out what's going on. I > know > > > this is inconvenient. We just haven't seen usage like this yet. > > > > > Christian > > > > > -- > > > Christian Hammond - chip...@chipx86.com > > > Review Board -http://www.review-board.org > > > VMware, Inc. -http://www.vmware.com > > > > > On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com> wrote: > > > > >> Any follow up on this? Any configuration guidance? > > > > >> We've moved to a 16 G + 4 cpu system and still have unacceptable > > >> response times (~30 seconds) for really large diffs with hundreds of > > >> review comments. > > > > >> If you can't reproduce this, I recommend you set the number of diff > > >> files per page from 20 to something really high (like 100) and then > > >> post a review with the same number of files deleted. For us, this puts > > >> a huge load on the local browser and the server VM during load. > > > > >> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote: > > >> > Thanks again. I'm really not that familiar with Apache 2 w/ regards > to > > >> > performance issues. At this time, I just have a more or less out-of- > > >> > the-box Apache2 setup on CentOS 5. Any recommendations here would be > > >> > hugely appreciated. Further answers inline below... > > > > >> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com> wrote: > > > > >> > > We should figure out first why your backend server is being hit so > > >> hard. We > > >> > > have over a thousand people at VMware submitting good sized diffs > and > > >> aren't > > >> > > hitting any of these issues. So a few more questions. Apologies if > > >> you've > > >> > > answered these before in other threads. > > > > >> > > 1) What MPM is your Apache using? > > > > >> > I'm using the default for httpd 2.2.3 CentOS. I believe this is > > >> > 'prefork'. > > > > >> > > 2) And is this fastcgi or mod_python? > > > > >> > mod_python (mod_python-3.2.8-3.1) > > > > >> > > 3) Are you absolutely sure Review Board is using your memcached > > >> server? Go > > >> > > in the admin dashboard and click Server Cache, then copy and paste > the > > >> info > > >> > > and paste it here. > > > > >> > Server Cache > > >> > Cache backend: django.core.cache.backends.memcached > > >> > Statistics > > >> > 127.0.0.1:11211 Memory usage: 45.7 MB > > >> > Keys in cache: 1767 of 1777 > > >> > Cache hits: 3996 of 7464: 53% > > >> > Cache misses: 3468 of 7464: 46% > > >> > Cache evictions: 0 > > >> > Cache traffic: 46.1 MB in, 103.8 MB out > > >> > Uptime: 53009 > > > > >> > > We might be able to do something smarter with deleted diffs, but > that > > >> would > > >> > > require a good bit of work, and we'd have to think carefully about > it. > > > > >> > > We talked about splitting up the reviews page, but decided it > wouldn't > > >> make > > >> > > much sense due to how it's ordered. We came up with a method for > > >> collapsing, > > >> > > but I'm probably holding off until after 1.0 to finish this, > because > > >> it > > >> > > needs a lot more thought and testing. > > > > >> > > Christian > > > > >> > > -- > > >> > > Christian Hammond - chip...@chipx86.com > > >> > > Review Board -http://www.review-board.org > > >> > > VMware, Inc. -http://www.vmware.com > > > > >> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <ciaom...@gmail.com> > > >> wrote: > > > > >> > > > Thanks for your response. Please see comments inline below... > > > > >> > > > On Mar 18, 11:35 am, Christian Hammond <chip...@chipx86.com> > wrote: > > >> > > > > Not immediately, no. > > > > >> > > > > I can't even determine right now whether we can do anything > about > > >> this. > > >> > > > > You're seeing crazy CPU loads that we're definitely not > seeing, > > >> and most > > >> > > > of > > >> > > > > this has to do with the browser itself. We're loading the > files > > >> > > > > asynchronously, but it appears that for whatever reason your > > >> browser is > > >> > > > > blocking until it's all loaded (if I'm understanding right). > > > > >> > > > Just for clarity, the page can sometimes load partially, but > since > > >> the > > >> > > > CPU is at 99% it's of little value. I don't get the CPU back > until > > >> the > > >> > > > page is fully loaded. > > > > >> > > > We now understand that our backend server is also getting maxed > out > > >> to > > >> > > > the point of failure. One request of this page demands 800 M of > > >> > > > virtual memory from Apache! Once the request is fully loaded, > 400 M > > >> of > > >> > > > that memory is freed again - but since we have 200 reviewboard > users > > >> > > > this isn't very scalable. > > > > >> > > > > I'm wondering > > >> > > > > if there's something with the configuration in your browsers > where > > >> you > > >> > > > work > > >> > > > > that is causing some of this. > > > > >> > > > I really don't think this is a browser configuration issue > because > > >> > > > users across many different OS and browsers are seeing this. I > see > > >> it > > >> > > > on default FireFox 3.0.7 and IE 7.0. Users have reported that > Safari > > >> > > > works the best, but I have not confirmed. FireFox 2 and IE 6 > users > > >> > > > have reported the same problem. > > > > >> > > > > You say this is happening on several different browsers? In > the > > >> case of > > >> > > > > Firefox, what extensions are loaded? > > > > >> > > > Just the default FireFox 3.0.7 install. > > > > >> > > > > We definitely would like to fix this. But we just don't know > > >> enough about > > >> > > > > what the problem is right now to determine if it's in our > control. > > > > >> > > > Apache must load up a huge amount of data in VM for these large > > >> diffs > > >> > > > (I believe this is especially true when diffs contain deleted > > >> files). > > >> > > > This isn't scalable across many users. Can the amount of data/ > > >> > > > filediffs/etc needed to load the page be scaled back further? > > >> Changing > > >> > > > the 'Paginate by' value from 20 to 3 has helped, but not enough > and > > >> it > > >> > > > hasn't helped the review page obviously. > > > > >> > > > Instead of showing all lines for deleted files can this be > loaded > > >> > > > later only on request? > > >> > > > Can the 'View Reviews' page be broken up or paginated as well? > > >> > > > I think these two suggestions would help us. > > > > >> > > > > Christian > > > > >> > > > > -- > > >> > > > > Christian Hammond - chip...@chipx86.com > > >> > > > > Review Board -http://www.review-board.org > > >> > > > > VMware, Inc. -http://www.vmware.com > > > > >> > > > > On Wed, Mar 18, 2009 at 9:54 AM, ciaomary <ciaom...@gmail.com > > > > >> wrote: > > > > >> > > > > > Can anything be done to fix the pegged CPU issue I've > reported > > >> in my > > >> > > > > > prior email? > > > > >> > > > > > On Mar 17, 9:38 am, ciaomary <ciaom...@gmail.com> wrote: > > >> > > > > > > The downloadable diff for the review is 1.6MB. > > > > >> > > > > > > The review has 138 different files (7 pages). > > > > >> > > > > > > Page 1 loads in ~1 minute in FireFox. The twenty files are > on > > >> average > > >> > > > > > > 38 K. (max=118K, min=2K). Here all files have edit changes > (no > > >> > > > deleted > > >> > > > > > > files.) > > > > >> > > > > > > Page 3 loads in ~3 minutes in FireFox. The twenty files > are on > > >> > > > average > > >> > > > > > > 25K . No major deviations from the average. Here 12 of the > > >> files have > > >> > > > > > > been deleted rather than just edited. > > > > >> > > > > > > There are hundreds of review comments on this review > across > > >> many > > >> > > > users > > >> > > > > > > and files. > > > > >> > > > > > > During page load, the CPU is pegged and the browser is > > >> unresponsive > > >> > > > > > > for a good 2-3 minutes. This is the cause every time, very > > >> > > > > > > reproducable. If you are patient the page finally loads > and > > >> the > > >> > > > system > > >> > > > > > > CPU returns to normal. > > > > >> > > > > > > FireBug shows an HTTPS GET request for each of the twenty > > >> files > > >> > > > during > > >> > > > > > > the slow down/cpu pegging. Each HTTPS GETS are taking > longer > > >> on page > > >> > > > > > > 3, with some taking up to 2.5 seconds (others < 50ms). > Since > > >> the is > > >> > > > > > > cpu is pegged the browser processing seems to be the cause > of > > >> the > > >> > > > > > > delay however. > > > > >> > > > > > > On Mar 16, 6:31 pm, Christian Hammond < > chip...@chipx86.com> > > >> wrote: > > > > >> > > > > > > > So it's really going to depend on the sizes of the > > >> individual files > > >> > > > > > > > themselves. If a file is by itself large enough to cause > a > > >> couple > > >> > > > meg > > >> > > > > > diff > > >> > > > > > > > to be generated, there's nothing we can do really do, > since > > >> that > > >> > > > boils > > >> > > > > > down > > >> > > > > > > > to too large a file to show in the browser. > > > > >> > > > > > > > In your page 3 example, how big are the files and the > diffs > > >> in > > >> > > > > > particular? > > > > >> > > > > > > > The page itself loads quick enough, right? It's just the > > >> diff > > >> > > > > > fragments? > > >> > > > > > > > Which ones are you finding takes a long time and how big > are > > >> those > > >> > > > > > changes? > > > > >> > > > > > > > Christian > > > > >> > > > > > > > -- > > >> > > > > > > > Christian Hammond - chip...@chipx86.com > > >> > > > > > > > Review Board -http://www.review-board.org > > >> > > > > > > > VMware, Inc. -http://www.vmware.com > > > > >> > > > > > > > On Mon, Mar 16, 2009 at 5:32 PM, mary < > ciaom...@gmail.com> > > >> wrote: > > > > >> > > > > > > > > On really large diffs, performance is very slow and > can > > >> crash the > > > > ... > > > > read more ยป- Hide quoted text - > > > > - Show quoted text - > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "reviewboard" group. To post to this group, send email to reviewboard@googlegroups.com To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~----------~----~----~----~------~----~------~--~---
