[users@httpd] Mod_proxy Slow After a Week
Hello all, We have: Apache 2.2.3 CentOS 5.5 x86_64 Splunk 5.0.2 I only know the basics but Apache has been serving us very well with the below config and only after about a week did pages refresh very slowly, up to a minute to reload sometimes. I tailed both httpd and splunkd logs and saw a consistent delay of 30s between proxy and origin server (both on same host) along with plenty of 304s, followed by what appeared to be some timeout and then slowly the 200s started coming back in. A restart of httpd cleared up the issue. I'm lost, please help. Could this be a caching problem? Thank you. SSLEngine on SSLCertificateFile /etc/httpd/conf/server.crt SSLCertificateKeyFile /etc/httpd/conf/server.key SSLProxyEngine On SSLCACertificateFile/etc/httpd/conf/gsoc.pem SSLProtocol all -SSLv2 SSLVerifyClient require SSLVerifyDepth 1 SSLOptions +StrictRequire RequestHeader set X-Remote-User %{REMOTE_USER}s ServerName dashboards.gsoc.co.za:443 ServerAdmin ad...@gsoc.co.za DocumentRoot/srv/http/gdf/ CustomLog /var/log/httpd/gdf/access combined ErrorLog/var/log/httpd/gdf/error LogLeveldebug ProxyRequests Off ProxyPreserveHost Off ProxyPass /gdf https://172.20.67.2:8000/gdf ProxyPassReverse /gdf https://172.20.67.2:8000/gdf SSLRequireSSL AllowOverride none AuthName "GDF" AuthType Basic AuthDigestProvider file AuthUserFile /etc/httpd/conf/passwd Require ssl-verify-client Require valid-user Require ssl Satisfy All SSLRequireSSL AuthName "GDF" AuthType Basic AuthDigestProvider file AuthUserFile /etc/httpd/conf/passwd Require ssl-verify-client Require valid-user Require ssl Satisfy All
Re: [users@httpd] Mod_proxy Slow After a Week
On 24 June 2013 17:49, David Guerra wrote: > You are receiving 304's? > That doesn't make sense. Maybe 504's which many times will point to a > load issue on the backend server. > > > On Mon, Jun 24, 2013 at 9:03 AM, Robert Gabriel wrote: > >> Hello all, >> >> We have: >> >> Apache 2.2.3 >> CentOS 5.5 x86_64 >> Splunk 5.0.2 >> >> I only know the basics but Apache has been serving us very well with the >> below config >> >> and only after about a week did pages refresh very slowly, up to a minute >> to reload >> >> sometimes. >> >> I tailed both httpd and splunkd logs and saw a consistent delay of 30s >> between proxy >> >> and origin server (both on same host) along with plenty of 304s, followed >> by what >> >> appeared to be some timeout and then slowly the 200s started coming back >> in. >> >> A restart of httpd cleared up the issue. >> >> I'm lost, please help. >> >> Could this be a caching problem? >> >> Thank you. >> >> >> SSLEngine on >> SSLCertificateFile /etc/httpd/conf/server.crt >> SSLCertificateKeyFile /etc/httpd/conf/server.key >> SSLProxyEngine On >> SSLCACertificateFile/etc/httpd/conf/gsoc.pem >> SSLProtocol all -SSLv2 >> SSLVerifyClient require >> SSLVerifyDepth 1 >> SSLOptions +StrictRequire >> >> RequestHeader set X-Remote-User %{REMOTE_USER}s >> >> ServerName dashboards.gsoc.co.za:443 >> ServerAdmin ad...@gsoc.co.za >> DocumentRoot/srv/http/gdf/ >> CustomLog /var/log/httpd/gdf/access combined >> ErrorLog/var/log/httpd/gdf/error >> LogLeveldebug >> >> ProxyRequests Off >> ProxyPreserveHost Off >> ProxyPass /gdf https://172.20.67.2:8000/gdf >> ProxyPassReverse /gdf https://172.20.67.2:8000/gdf >> >> >> SSLRequireSSL >> AllowOverride none >> AuthName "GDF" >> AuthType Basic >> AuthDigestProvider file >> AuthUserFile /etc/httpd/conf/passwd >> Require ssl-verify-client >> Require valid-user >> Require ssl >> Satisfy All >> >> >> >> SSLRequireSSL >> AuthName "GDF" >> AuthType Basic >> AuthDigestProvider file >> AuthUserFile /etc/httpd/conf/passwd >> Require ssl-verify-client >> Require valid-user >> Require ssl >> Satisfy All >> >> >> >> > > > -- > Take Care, > > David Guerra > http://frustratedtech.com/ > 304s for sure, no doubt.
Re: [users@httpd] Mod_proxy Slow After a Week
On 24 June 2013 18:27, Tom Evans wrote: > On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel > wrote: > > Hello all, > > > > We have: > > > > Apache 2.2.3 > > CentOS 5.5 x86_64 > > Splunk 5.0.2 > > > > I only know the basics but Apache has been serving us very well with the > > below config > > > > and only after about a week did pages refresh very slowly, up to a > minute to > > reload > > > > sometimes. > > > > I tailed both httpd and splunkd logs and saw a consistent delay of 30s > > between proxy > > > > and origin server (both on same host) along with plenty of 304s, > followed by > > what > > > > appeared to be some timeout and then slowly the 200s started coming back > in. > > > > A restart of httpd cleared up the issue. > > > > I'm lost, please help. > > > > Could this be a caching problem? > > > > Thank you. > > > > > > SSLEngine on > > SSLCertificateFile /etc/httpd/conf/server.crt > > SSLCertificateKeyFile /etc/httpd/conf/server.key > > SSLProxyEngine On > > SSLCACertificateFile/etc/httpd/conf/gsoc.pem > > SSLProtocol all -SSLv2 > > SSLVerifyClient require > > SSLVerifyDepth 1 > > SSLOptions +StrictRequire > > > > RequestHeader set X-Remote-User %{REMOTE_USER}s > > > > ServerName dashboards.gsoc.co.za:443 > > ServerAdmin ad...@gsoc.co.za > > DocumentRoot/srv/http/gdf/ > > CustomLog /var/log/httpd/gdf/access combined > > ErrorLog/var/log/httpd/gdf/error > > LogLeveldebug > > > > ProxyRequests Off > > ProxyPreserveHost Off > > ProxyPass /gdf https://172.20.67.2:8000/gdf > > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf > > > > > > SSLRequireSSL > > AllowOverride none > > AuthName "GDF" > > AuthType Basic > > AuthDigestProvider file > > AuthUserFile /etc/httpd/conf/passwd > > Require ssl-verify-client > > Require valid-user > > Require ssl > > Satisfy All > > > > > > > > SSLRequireSSL > > AuthName "GDF" > > AuthType Basic > > AuthDigestProvider file > > AuthUserFile /etc/httpd/conf/passwd > > Require ssl-verify-client > > Require valid-user > > Require ssl > > Satisfy All > > > > > > 30 seconds is the length of the default timeout in apache. > Unfortunately, that timeout is used in all sorts of cases, so it does > not tell us what is timing out. > > As a rank guess, I would be going for DNS timeout myself. Do you have > HostnameLookups set to "On" or "Double", or using host names in ACLs? > > Cheers > > Tom > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > > We are not doing any hostname ACLs. Forgive my limited knowledge, I did RTFM before replying to make sure I understood the intended config and no, we are not doing any. "HostnameLookups Off" How come a restart "fixes" the problem? Thank you.
Re: [users@httpd] Mod_proxy Slow After a Week
On 26 June 2013 09:59, Thomas Eckert wrote: > This is interesting. I'm seeing similar events at one of my reverse proxy. > After fixing a DNS infrastructure problem I started seeing lots of 304s > with processing times well above 5 seconds. For header-only requests that's > a lot of time. If there is any news here please let us know. At the very > least it would help me. Will do. So far so good. Thank you.
Re: [users@httpd] Mod_proxy Slow After a Week
On 26 June 2013 12:40, Tom Evans wrote: > On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel > wrote: > > Hello all, > > > > We have: > > > > Apache 2.2.3 > > CentOS 5.5 x86_64 > > Splunk 5.0.2 > > Sorry, I didn't read your original post. Please upgrade Apache to > 2.2.24 and re-test. > Thank you, will do. I'll keep an eye and advise.
Re: [users@httpd] Mod_proxy Slow After a Week
On 24 June 2013 18:27, Tom Evans wrote: On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel wrote: > > Hello all, > > > > We have: > > > > Apache 2.2.3 > > CentOS 5.5 x86_64 > > Splunk 5.0.2 > > > > I only know the basics but Apache has been serving us very well with the > > below config > > > > and only after about a week did pages refresh very slowly, up to a > minute to > > reload > > > > sometimes. > > > > I tailed both httpd and splunkd logs and saw a consistent delay of 30s > > between proxy > > > > and origin server (both on same host) along with plenty of 304s, > followed by > > what > > > > appeared to be some timeout and then slowly the 200s started coming back > in. > > > > A restart of httpd cleared up the issue. > > > > I'm lost, please help. > > > > Could this be a caching problem? > > > > Thank you. > > > > > > SSLEngine on > > SSLCertificateFile /etc/httpd/conf/server.crt > > SSLCertificateKeyFile /etc/httpd/conf/server.key > > SSLProxyEngine On > > SSLCACertificateFile/etc/httpd/conf/gsoc.pem > > SSLProtocol all -SSLv2 > > SSLVerifyClient require > > SSLVerifyDepth 1 > > SSLOptions +StrictRequire > > > > RequestHeader set X-Remote-User %{REMOTE_USER}s > > > > ServerName dashboards.gsoc.co.za:443 > > ServerAdmin ad...@gsoc.co.za > > DocumentRoot/srv/http/gdf/ > > CustomLog /var/log/httpd/gdf/access combined > > ErrorLog/var/log/httpd/gdf/error > > LogLeveldebug > > > > ProxyRequests Off > > ProxyPreserveHost Off > > ProxyPass /gdf https://172.20.67.2:8000/gdf > > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf > > > > > > SSLRequireSSL > > AllowOverride none > > AuthName "GDF" > > AuthType Basic > > AuthDigestProvider file > > AuthUserFile /etc/httpd/conf/passwd > > Require ssl-verify-client > > Require valid-user > > Require ssl > > Satisfy All > > > > > > > > SSLRequireSSL > > AuthName "GDF" > > AuthType Basic > > AuthDigestProvider file > > AuthUserFile /etc/httpd/conf/passwd > > Require ssl-verify-client > > Require valid-user > > Require ssl > > Satisfy All > > > > > > 30 seconds is the length of the default timeout in apache. > Unfortunately, that timeout is used in all sorts of cases, so it does > not tell us what is timing out. > > As a rank guess, I would be going for DNS timeout myself. Do you have > HostnameLookups set to "On" or "Double", or using host names in ACLs >Sorry, I didn't read your original post. Please upgrade Apache to >2.2.24 and re-test. We upgraded Apache and installed CentOS-6.4. Problem gone.