Re: Diagnosing Ask Fedora sluggishness
https://fedorahosted.org/fedora-infrastructure/ticket/4013#comment:2 Default setup - 21 requests, 657kb, 1.04 s, http://tools.pingdom.com/fpt/#!/3R3vS/http://fedora.hackalyst.info/questions/ Optimized setup - 15 requestsm 253kb, 708 ms , http://tools.pingdom.com/fpt/#!/eNP7yA/http://fedora.hackalyst.info/questions/ Both from NYC (This is the latest askbot 0.7.49 install) Changes made : Added another domain for static resources . This reduces queue blocking. Enabled js/css cache and minimization in django, this reduces number of resource calls and their sizes. Enabled gzip in apache. gzip in django is very restrictive and doesn't serve our purpose here. Askbot / django changes are all in settings.py only. To Do: Figure out way to split js, css and images to various domains. This will improve speed further by cutting down wait time by half. regards Anshu Prateek On Mon, Sep 9, 2013 at 1:31 PM, Anshu Prateek wrote: > To be improved @ ask.fp.o > > > Grade F on Configure entity tags (ETags) > There are 26 components with misconfigured ETags > > upstream askbot is good on this. > > Grade A on Configure entity tags (ETags) > > Other than this, everything else looks like can be fixed at upstream. > > One major difference against upstream is ask.fpo is running on https. > > The wait time on the initial connect itself is about 0.5 sec. As much as I > see, NY is served from a VM at UNC vs a linode for the askbot. So I guess > we will have to do more tweaking on our end even just to be at par with the > upstream if not better. > > > > > On Mon, Sep 9, 2013 at 5:32 AM, Ankur Sinha wrote: > >> Hi Anshu, Kevin, >> >> On Sun, 2013-09-08 at 10:07 -0600, Kevin Fenzi wrote: >> > On Sun, 8 Sep 2013 14:19:20 +0530 >> > Anshu Prateek wrote: >> > >> > > The generic yslow grades suggest couple of improvements: >> > > >> > > Grade F on Add Expires headers >> > > >> > > There are 55 static components without a far-future expiration date. >> > > Grade F on Configure entity tags (ETags) There are 27 components with >> > > misconfigured ETags >> > > >> > > Grade E on Use cookie-free domainsThere are 9 components that are not >> > > cookie-free >> > > >> > > And some more. >> > > >> > > All of these for https://ask.fedoraproject.org/questions/ >> >> Can you tell if these are issues limited to our deployment, or if they >> are things upstream needs to correct, in which case we can go ahead and >> file bugs? >> >> > >> > Yeah, askbot is using our haproxy/varnish setup as well as memcached, so >> > if upstream can tweak things to be more caching friendly that would help >> > for sure. >> >> I compared the pingdom tests for our askbot instance, against the one >> that upstream has deployed: >> >> http://tools.pingdom.com/fpt/#!/bFd18x/http://askbot.org/en/questions/ >> >> >> http://tools.pingdom.com/fpt/#!/dmLsqz/http://ask.fedoraproject.org/questions/ >> >> Upstream's deployment takes 1.07 seconds, while ours takes 3.07. Would >> this imply that we can at least tweak our deployment to perform as quick >> as upstream's? >> >> (Upstream's comment posting etc. doesn't lag at all either. The comment >> is posted as soon as one clicks "post".) >> >> > >> > Actually in this case you might want to look at the django level one? >> > modules/askbot/templates/settings.erb: >> > >> > MIDDLEWARE_CLASSES = ( >> > #'django.middleware.gzip.GZipMiddleware', >> > >> > There may also be some other caching stuff there we should or shouldn't >> > enable... not sure anyone has fully investigated all those. >> > >> >> I'll look into these, see if there's one we can use. >> >> -- >> Thanks, >> Warm regards, >> Ankur (FranciscoD) >> >> http://fedoraproject.org/wiki/User:Ankursinha >> >> Join Fedora! Come talk to us! >> http://fedoraproject.org/wiki/Fedora_Join_SIG >> >> >> ___ >> infrastructure mailing list >> infrastructure@lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/infrastructure >> > > ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
To be improved @ ask.fp.o Grade F on Configure entity tags (ETags) There are 26 components with misconfigured ETags upstream askbot is good on this. Grade A on Configure entity tags (ETags) Other than this, everything else looks like can be fixed at upstream. One major difference against upstream is ask.fpo is running on https. The wait time on the initial connect itself is about 0.5 sec. As much as I see, NY is served from a VM at UNC vs a linode for the askbot. So I guess we will have to do more tweaking on our end even just to be at par with the upstream if not better. On Mon, Sep 9, 2013 at 5:32 AM, Ankur Sinha wrote: > Hi Anshu, Kevin, > > On Sun, 2013-09-08 at 10:07 -0600, Kevin Fenzi wrote: > > On Sun, 8 Sep 2013 14:19:20 +0530 > > Anshu Prateek wrote: > > > > > The generic yslow grades suggest couple of improvements: > > > > > > Grade F on Add Expires headers > > > > > > There are 55 static components without a far-future expiration date. > > > Grade F on Configure entity tags (ETags) There are 27 components with > > > misconfigured ETags > > > > > > Grade E on Use cookie-free domainsThere are 9 components that are not > > > cookie-free > > > > > > And some more. > > > > > > All of these for https://ask.fedoraproject.org/questions/ > > Can you tell if these are issues limited to our deployment, or if they > are things upstream needs to correct, in which case we can go ahead and > file bugs? > > > > > Yeah, askbot is using our haproxy/varnish setup as well as memcached, so > > if upstream can tweak things to be more caching friendly that would help > > for sure. > > I compared the pingdom tests for our askbot instance, against the one > that upstream has deployed: > > http://tools.pingdom.com/fpt/#!/bFd18x/http://askbot.org/en/questions/ > > > http://tools.pingdom.com/fpt/#!/dmLsqz/http://ask.fedoraproject.org/questions/ > > Upstream's deployment takes 1.07 seconds, while ours takes 3.07. Would > this imply that we can at least tweak our deployment to perform as quick > as upstream's? > > (Upstream's comment posting etc. doesn't lag at all either. The comment > is posted as soon as one clicks "post".) > > > > > Actually in this case you might want to look at the django level one? > > modules/askbot/templates/settings.erb: > > > > MIDDLEWARE_CLASSES = ( > > #'django.middleware.gzip.GZipMiddleware', > > > > There may also be some other caching stuff there we should or shouldn't > > enable... not sure anyone has fully investigated all those. > > > > I'll look into these, see if there's one we can use. > > -- > Thanks, > Warm regards, > Ankur (FranciscoD) > > http://fedoraproject.org/wiki/User:Ankursinha > > Join Fedora! Come talk to us! > http://fedoraproject.org/wiki/Fedora_Join_SIG > > > ___ > infrastructure mailing list > infrastructure@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/infrastructure > ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
Hi Anshu, Kevin, On Sun, 2013-09-08 at 10:07 -0600, Kevin Fenzi wrote: > On Sun, 8 Sep 2013 14:19:20 +0530 > Anshu Prateek wrote: > > > The generic yslow grades suggest couple of improvements: > > > > Grade F on Add Expires headers > > > > There are 55 static components without a far-future expiration date. > > Grade F on Configure entity tags (ETags) There are 27 components with > > misconfigured ETags > > > > Grade E on Use cookie-free domainsThere are 9 components that are not > > cookie-free > > > > And some more. > > > > All of these for https://ask.fedoraproject.org/questions/ Can you tell if these are issues limited to our deployment, or if they are things upstream needs to correct, in which case we can go ahead and file bugs? > > Yeah, askbot is using our haproxy/varnish setup as well as memcached, so > if upstream can tweak things to be more caching friendly that would help > for sure. I compared the pingdom tests for our askbot instance, against the one that upstream has deployed: http://tools.pingdom.com/fpt/#!/bFd18x/http://askbot.org/en/questions/ http://tools.pingdom.com/fpt/#!/dmLsqz/http://ask.fedoraproject.org/questions/ Upstream's deployment takes 1.07 seconds, while ours takes 3.07. Would this imply that we can at least tweak our deployment to perform as quick as upstream's? (Upstream's comment posting etc. doesn't lag at all either. The comment is posted as soon as one clicks "post".) > > Actually in this case you might want to look at the django level one? > modules/askbot/templates/settings.erb: > > MIDDLEWARE_CLASSES = ( > #'django.middleware.gzip.GZipMiddleware', > > There may also be some other caching stuff there we should or shouldn't > enable... not sure anyone has fully investigated all those. > I'll look into these, see if there's one we can use. -- Thanks, Warm regards, Ankur (FranciscoD) http://fedoraproject.org/wiki/User:Ankursinha Join Fedora! Come talk to us! http://fedoraproject.org/wiki/Fedora_Join_SIG signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
On Sun, 8 Sep 2013 14:19:20 +0530 Anshu Prateek wrote: > The generic yslow grades suggest couple of improvements: > > Grade F on Add Expires headers > > There are 55 static components without a far-future expiration date. > Grade F on Configure entity tags (ETags) There are 27 components with > misconfigured ETags > > Grade E on Use cookie-free domainsThere are 9 components that are not > cookie-free > > And some more. > > All of these for https://ask.fedoraproject.org/questions/ Yeah, askbot is using our haproxy/varnish setup as well as memcached, so if upstream can tweak things to be more caching friendly that would help for sure. > > >>Oh, and can we enable gzip compression on the ask hosts? (I don't > > >>know > > how to do it yet.) > > > > Its in the apache config file. > > > > http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#enable Actually in this case you might want to look at the django level one? modules/askbot/templates/settings.erb: MIDDLEWARE_CLASSES = ( #'django.middleware.gzip.GZipMiddleware', There may also be some other caching stuff there we should or shouldn't enable... not sure anyone has fully investigated all those. kevin signature.asc Description: PGP signature ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
On 08/09/13 09:49, Anshu Prateek wrote: > Also, I was about to send a similar mail about site being slow for the > badges. Are these two sharing the hosts? I asked Ralph about Badge slowness the other day. Ralph wrote: > there are fixes in git -- some merged, some still under review -- > that address the speed issue. > I think this is the last one standing - > https://github.com/fedora-infra/tahrir/pull/211 Kind regards, -- Jamie Nguyen ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
The generic yslow grades suggest couple of improvements: Grade F on Add Expires headers There are 55 static components without a far-future expiration date. Grade F on Configure entity tags (ETags) There are 27 components with misconfigured ETags Grade E on Use cookie-free domainsThere are 9 components that are not cookie-free And some more. All of these for https://ask.fedoraproject.org/questions/ Also, I was about to send a similar mail about site being slow for the badges. Are these two sharing the hosts? On Sun, Sep 8, 2013 at 2:09 PM, Anshu Prateek wrote: > >>Oh, and can we enable gzip compression on the ask hosts? (I don't know > how to do it yet.) > > Its in the apache config file. > > http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#enable > > > On Sun, Sep 8, 2013 at 1:58 PM, Ankur Sinha wrote: > >> On Sun, 2013-09-08 at 08:08 +0530, Aditya Patawari wrote: >> > Hi Ankur, >> >> Hi Aditya, >> >> > >> > From what I see, after comparing the time used to load each asset, >> > there are a couple of java scripts, util.js and less.min.js, on Ask >> > Fedora, which takes quite a bit to load. Could it be because of this? >> >> I'm really not sure. I did some random checking and found a few hints, >> but nothing concret: >> >> >> http://tools.pingdom.com/fpt/dmLsqz/http://ask.fedoraproject.org/questions/ >> >> http://askbot.org/en/question/3172/how-to-improve-the-speed-of-askbot/ >> >> > >> > Although there seems to be a varnish cache to deliver it quicker but >> > somehow the load time runs in many seconds. >> >> It probably is the JS, since it's the commenting and voting etc part >> that takes time, and this seems to be JS controlled. >> >> http://askbot.org/en/question/8514/why-does-askbot-require-javascript/ >> >> Any ideas on how we could speed this up? >> >> Oh, and can we enable gzip compression on the ask hosts? (I don't know >> how to do it yet.) >> -- >> Thanks, >> Warm regards, >> Ankur (FranciscoD) >> >> http://fedoraproject.org/wiki/User:Ankursinha >> >> Join Fedora! Come talk to us! >> http://fedoraproject.org/wiki/Fedora_Join_SIG >> >> >> ___ >> infrastructure mailing list >> infrastructure@lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/infrastructure >> > > ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
On Sun, 2013-09-08 at 14:09 +0530, Anshu Prateek wrote: > Its in the apache config file. > > http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#enable > I figured. Not the big issue here though. -- Thanks, Warm regards, Ankur (FranciscoD) http://fedoraproject.org/wiki/User:Ankursinha Join Fedora! Come talk to us! http://fedoraproject.org/wiki/Fedora_Join_SIG signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
>>Oh, and can we enable gzip compression on the ask hosts? (I don't know how to do it yet.) Its in the apache config file. http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#enable On Sun, Sep 8, 2013 at 1:58 PM, Ankur Sinha wrote: > On Sun, 2013-09-08 at 08:08 +0530, Aditya Patawari wrote: > > Hi Ankur, > > Hi Aditya, > > > > > From what I see, after comparing the time used to load each asset, > > there are a couple of java scripts, util.js and less.min.js, on Ask > > Fedora, which takes quite a bit to load. Could it be because of this? > > I'm really not sure. I did some random checking and found a few hints, > but nothing concret: > > http://tools.pingdom.com/fpt/dmLsqz/http://ask.fedoraproject.org/questions/ > > http://askbot.org/en/question/3172/how-to-improve-the-speed-of-askbot/ > > > > > Although there seems to be a varnish cache to deliver it quicker but > > somehow the load time runs in many seconds. > > It probably is the JS, since it's the commenting and voting etc part > that takes time, and this seems to be JS controlled. > > http://askbot.org/en/question/8514/why-does-askbot-require-javascript/ > > Any ideas on how we could speed this up? > > Oh, and can we enable gzip compression on the ask hosts? (I don't know > how to do it yet.) > -- > Thanks, > Warm regards, > Ankur (FranciscoD) > > http://fedoraproject.org/wiki/User:Ankursinha > > Join Fedora! Come talk to us! > http://fedoraproject.org/wiki/Fedora_Join_SIG > > > ___ > infrastructure mailing list > infrastructure@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/infrastructure > ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
On Sun, 2013-09-08 at 18:28 +1000, Ankur Sinha wrote: > http://tools.pingdom.com/fpt/dmLsqz/http://ask.fedoraproject.org/questions/ The flags seem to take quite a while, and yes, the JS too. -- Thanks, Warm regards, Ankur (FranciscoD) http://fedoraproject.org/wiki/User:Ankursinha Join Fedora! Come talk to us! http://fedoraproject.org/wiki/Fedora_Join_SIG signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
On Sun, 2013-09-08 at 08:08 +0530, Aditya Patawari wrote: > Hi Ankur, Hi Aditya, > > From what I see, after comparing the time used to load each asset, > there are a couple of java scripts, util.js and less.min.js, on Ask > Fedora, which takes quite a bit to load. Could it be because of this? I'm really not sure. I did some random checking and found a few hints, but nothing concret: http://tools.pingdom.com/fpt/dmLsqz/http://ask.fedoraproject.org/questions/ http://askbot.org/en/question/3172/how-to-improve-the-speed-of-askbot/ > > Although there seems to be a varnish cache to deliver it quicker but > somehow the load time runs in many seconds. It probably is the JS, since it's the commenting and voting etc part that takes time, and this seems to be JS controlled. http://askbot.org/en/question/8514/why-does-askbot-require-javascript/ Any ideas on how we could speed this up? Oh, and can we enable gzip compression on the ask hosts? (I don't know how to do it yet.) -- Thanks, Warm regards, Ankur (FranciscoD) http://fedoraproject.org/wiki/User:Ankursinha Join Fedora! Come talk to us! http://fedoraproject.org/wiki/Fedora_Join_SIG signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Diagnosing Ask Fedora sluggishness
Hi Ankur, From what I see, after comparing the time used to load each asset, there are a couple of java scripts, util.js and less.min.js, on Ask Fedora, which takes quite a bit to load. Could it be because of this? Although there seems to be a varnish cache to deliver it quicker but somehow the load time runs in many seconds. Aditya Patawari http://blog.adityapatawari.com/ https://fedoraproject.org/wiki/User:Adimania India On Sun, Sep 8, 2013 at 7:52 AM, Ankur Sinha wrote: > Hi, > > Ask Fedora is still much slower than other askbot instances that I've > used. Try askbot.org/en/questions, for example: comments are posted as > soon as the user clicks. On Ask Fedora, there's quite a lag for posting > comments/answers. > > We shifted to django-postoffice to somewhat decouple the main > application from the mail sending, but I'm not sure if this made a > difference really. > > The askbot log only has errors about icons: > http://fpaste.org/37879/60667413/ > > I see quite a few errors in the httpd/error_log but I'm not sure if > they're responsible for the sluggishness either. > > I'm not sure where else I should look. Would someone have hints on how > to go about diagnosing this? > -- > Thanks, > Warm regards, > Ankur (FranciscoD) > > http://fedoraproject.org/wiki/User:Ankursinha > > Join Fedora! Come talk to us! > http://fedoraproject.org/wiki/Fedora_Join_SIG > > > ___ > infrastructure mailing list > infrastructure@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/infrastructure ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
