Hi Ze, The spikes every 5 minutes are interesting. Sounds like a cronjob or something, perhaps? Are you using search indexing?
What are you using for the database? Remind me what version of RB you guys are using? - Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.reviewboard.org Beanbag, Inc. - http://www.beanbaginc.com On Fri, Mar 7, 2014 at 3:36 PM, Ze Lin Xiao <ilacknormal...@gmail.com>wrote: > Sorry for the late response. I missed this reply. > For Apache settings the worker and prefork configurations are the exact > same between the two vms: > > # prefork MPM > # StartServers: number of server processes to start > # MinSpareServers: minimum number of server processes which are kept spare > # MaxSpareServers: maximum number of server processes which are kept spare > # ServerLimit: maximum value for MaxClients for the lifetime of the server > # MaxClients: maximum number of server processes allowed to start > # MaxRequestsPerChild: maximum number of requests a server process serves > <IfModule prefork.c> > StartServers 8 > MinSpareServers 5 > MaxSpareServers 20 > ServerLimit 256 > MaxClients 256 > MaxRequestsPerChild 100 > </IfModule> > > # worker MPM > # StartServers: initial number of server processes to start > # MaxClients: maximum number of simultaneous client connections > # MinSpareThreads: minimum number of worker threads which are kept spare > # MaxSpareThreads: maximum number of worker threads which are kept spare > # ThreadsPerChild: constant number of worker threads in each server process > # MaxRequestsPerChild: maximum number of requests a server process serves > <IfModule worker.c> > StartServers 2 > MaxClients 150 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadsPerChild 25 > MaxRequestsPerChild 0 > </IfModule> > > > *Symptoms from our old production vm:* > Previously, only about 2-3 times a day at random times, we would get a > build of Apache processes that would hit the server at the same time, which > results in the load average on top going up to 100-200 in the worse case > scenario. During this time, any operations done in the website are > extremely slow and often times users will report not receiving an email > after a publish. Since then, we've increased the sendmail Queue and Refuse > limits from their default values of (12 and 15) to (20 and 150) > respectively. > The stats on this vm were: > - 10 GB RAM > - 23 GB Swap > - 4 Cores > - RHEL 5.3 > - Red Hat Enterprise Linux Server release 5.3 (Tikanga) > - Server version: Apache/2.2.3 > > *Symptoms on our new production vm:* > We moved the production to RHEL6.4 as recommended by our IT team and have > since been noticing that we we get these stalled processes more often and > users tend to notice the performance hits much more. Another odd thing > that we noticed from our performance monitoring tool Zenoss is that the IO > spikes on writes every 5 minutes, which was on the case previously > (screenshots attached). > > The stats on this vm were: > - 16 GB RAM > - 4 GB Swap > - 4 Cores > - Red Hat Enterprise Linux Server release 6.4 (Santiago) > - Server version: Apache/2.2.15 (Unix) > > We're trying a lot of different things on our end, but if you have any > ideas or if anyone has seen this issue, it would help. > > > <https://lh4.googleusercontent.com/-L5GVIinnSbo/UxpXxkdzJDI/AAAAAAAACkg/Aq9DYWW6AXs/s1600/Screen+Shot+2014-03-07+at+3.35.13+PM.png> > > > <https://lh6.googleusercontent.com/-7FD3HifikjY/UxpXjy3QUwI/AAAAAAAACkY/BFO-tGDUwoE/s1600/Screen+Shot+2014-03-07+at+3.33.09+PM.png> > > > > Ze > > On Thursday, March 6, 2014 8:21:00 PM UTC-8, Christian Hammond wrote: > >> Okay, well, I was hoping it'd be simple :) >> >> Can you give me some examples of operations that are very slow, and >> operations that remain fast? Or does everything basically slow to a grind? >> >> How do the Apache settings (worker vs prefork, and their config) compare >> between installs? >> >> Christian >> >> >> On Thursday, March 6, 2014, Ze Xiao <ilackno...@gmail.com> wrote: >> >>> Thanks for the quick reply. Yes, memcached is running. Here is what I >>> see from the Admin Server Cache page >>> >>> I've got it running on two different vms, which I've obfuscated as "VM1" >>> and "VM2" >>> >>> SERVER CACHE >>> Cache backend: >>> >>> django.core.cache.backends.memcached.CacheClass >>> vm1 >>> Memory usage: >>> >>> 1.8 GB >>> Keys in cache: >>> >>> 61079 of 257077 >>> Cache hits: >>> >>> 5289571 of 5458860: 96% >>> Cache misses: >>> >>> 169289 of 5458860: 3% >>> Cache evictions: >>> >>> 139881 >>> Cache traffic: >>> >>> 10.2 GB in, 27.9 GB out >>> Uptime: >>> >>> 3683047 seconds >>> vm2 >>> Memory usage: >>> >>> 1.8 GB >>> Keys in cache: >>> >>> 54978 of 401980 >>> Cache hits: >>> >>> 5999634 of 6277198: 95% >>> Cache misses: >>> >>> 277564 of 6277198: 4% >>> Cache evictions: >>> >>> 307751 >>> Cache traffic: >>> >>> 16.8 GB in, 26.2 GB out >>> Uptime: >>> >>> 938019 seconds >>> >>> >>> On Thu, Mar 6, 2014 at 5:20 PM, Christian Hammond >>> <chip...@chipx86.com>wrote: >>> >>> Hi Ze, >>> >>> Those warnings are probably unrelated. >>> >>> I want to get a better sense of the performance problems. First thing I >>> want to check is that your server is properly accessing and using >>> memcached. If you log into the admin UI, do you see any stats on memcached, >>> and any keys stored in the cache? >>> >>> Christian >>> >>> -- >>> Christian Hammond - chip...@chipx86.com >>> Review Board - http://www.reviewboard.org >>> Beanbag, Inc. - http://www.beanbaginc.com >>> >>> >>> On Thu, Mar 6, 2014 at 4:52 PM, Ze Lin Xiao <ilacknormal...@gmail.com>wrote: >>> >>> Hi Christian, >>> >>> We're facing some pretty bad performance issues on our production system >>> after we moved our application to a different vm with RHEL6.4. >>> >>> We notice that our performance issues occur especially when the log >>> shows this: >>> [Fri Mar 07 00:18:19 2014] [error] /opt/software/lib/python2.7/ >>> site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg/Crypto/Util/number.py:57: >>> PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using >>> libgmp >= 5 to avoid timing attack vulnerability. >>> >>> However, it is important to note that we've seen these warning issues >>> for the last 1.5 years, so I doubt it has to do with it. Nonetheless, do >>> you know what specific operations one could do to trigger this warning? I'm >>> trying to see if I can reproduce the performance spikes. >>> >>> Thanks, >>> Ze >>> >>> On Wednesday, February 6, 2013 12:22:49 AM UTC-8, Christian Hammond >>> wrote: >>> >>> Hi Chuck, >>> >>> Sorry for failing to respond to the previous e-mail. Missed it. >>> >>> I haven't seen that particular warning before. It'll probably have a log >>> entry any time pycrypto is imported. What distro/version are you using? >>> Sounds like maybe it's an older one? You may need to hand-upgrade libgmp, >>> I'm not sure. >>> >>> From your previous e-mail: >>> >>> Doing a site backup never hurts, but generally isn't important. >>> >>> Review Board won't delete any files. At most, it'd add some new >>> directories and tell you to change permissions, but I don't think we've >>> done that since 1.5. We have provided instructions on other sorts of manual >>> updates that need to be made, though. >>> >>> We don't have any documentation right now on p4python's SSL support. >>> This is only needed if you're using SSL-backed Perforce repositories. It's >>> unfortunately not something we can automate well right now, but >>> essentially, you'd have to install OpenSSL 1.0.1 on your distro and install >>> its development package (I don't know if newer versions work -- hopefully >>> other 1.0.x releases do). You'd then need to manually compile/install >>> p4python. Yes, it's a pain, but it's something Perforce will need to make >>> easier for us. >>> >>> From the e-mail you just posted while I was replying to this, you'd need >>> to check the reviewboard.log file and see what error it's reporting before >>> I can say what happened. >>> >>> Christian >>> >>> -- >>> Christian Hammond - chi...@chipx86.com >>> >>> Review Board - http://www.reviewboard.org >>> VMware, Inc. - http://www.vmware.com >>> >>> On Feb 6, 2013, at 12:10 AM, chuck j <cjerr...@gmail.com> wrote: >>> >>> Hi Christian, >>> >>> I would like to thank you for your response about upgrade. >>> >>> I went through with your comments and i was able to bring my server to >>> 1.7.4. >>> >>> Also also want to bring to your notice regarding below warning i got >>> after while upgrading my site. >>> >>> /usr/local/lib/python2.7/site- >>> >>> -- >>> Ze Lin Xiao >>> >>> -- >>> Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ >>> --- >>> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ >>> --- >>> Happy user? Let us know at http://www.reviewboard.org/users/ >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "reviewboard" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to reviewboard+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> -- >> Christian Hammond - chi...@chipx86.com >> >> Review Board - http://www.reviewboard.org >> Beanbag, Inc. - http://www.beanbaginc.com >> >> -- > Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ > --- > Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ > --- > Happy user? Let us know at http://www.reviewboard.org/users/ > --- > You received this message because you are subscribed to the Google Groups > "reviewboard" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to reviewboard+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "reviewboard" group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
