[users@httpd] Mod_proxy Slow After a Week
Hello all,
We have:
Apache 2.2.3
CentOS 5.5 x86_64
Splunk 5.0.2
I only know the basics but Apache has been serving us very well with the
below config
and only after about a week did pages refresh very slowly, up to a minute
to reload
sometimes.
I tailed both httpd and splunkd logs and saw a consistent delay of 30s
between proxy
and origin server (both on same host) along with plenty of 304s, followed
by what
appeared to be some timeout and then slowly the 200s started coming back in.
A restart of httpd cleared up the issue.
I'm lost, please help.
Could this be a caching problem?
Thank you.
SSLEngine on
SSLCertificateFile /etc/httpd/conf/server.crt
SSLCertificateKeyFile /etc/httpd/conf/server.key
SSLProxyEngine On
SSLCACertificateFile/etc/httpd/conf/gsoc.pem
SSLProtocol all -SSLv2
SSLVerifyClient require
SSLVerifyDepth 1
SSLOptions +StrictRequire
RequestHeader set X-Remote-User %{REMOTE_USER}s
ServerName dashboards.gsoc.co.za:443
ServerAdmin ad...@gsoc.co.za
DocumentRoot/srv/http/gdf/
CustomLog /var/log/httpd/gdf/access combined
ErrorLog/var/log/httpd/gdf/error
LogLeveldebug
ProxyRequests Off
ProxyPreserveHost Off
ProxyPass /gdf https://172.20.67.2:8000/gdf
ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
SSLRequireSSL
AllowOverride none
AuthName "GDF"
AuthType Basic
AuthDigestProvider file
AuthUserFile /etc/httpd/conf/passwd
Require ssl-verify-client
Require valid-user
Require ssl
Satisfy All
SSLRequireSSL
AuthName "GDF"
AuthType Basic
AuthDigestProvider file
AuthUserFile /etc/httpd/conf/passwd
Require ssl-verify-client
Require valid-user
Require ssl
Satisfy All
Re: [users@httpd] Mod_proxy Slow After a Week
On 24 June 2013 17:49, David Guerra wrote:
> You are receiving 304's?
> That doesn't make sense. Maybe 504's which many times will point to a
> load issue on the backend server.
>
>
> On Mon, Jun 24, 2013 at 9:03 AM, Robert Gabriel wrote:
>
>> Hello all,
>>
>> We have:
>>
>> Apache 2.2.3
>> CentOS 5.5 x86_64
>> Splunk 5.0.2
>>
>> I only know the basics but Apache has been serving us very well with the
>> below config
>>
>> and only after about a week did pages refresh very slowly, up to a minute
>> to reload
>>
>> sometimes.
>>
>> I tailed both httpd and splunkd logs and saw a consistent delay of 30s
>> between proxy
>>
>> and origin server (both on same host) along with plenty of 304s, followed
>> by what
>>
>> appeared to be some timeout and then slowly the 200s started coming back
>> in.
>>
>> A restart of httpd cleared up the issue.
>>
>> I'm lost, please help.
>>
>> Could this be a caching problem?
>>
>> Thank you.
>>
>>
>> SSLEngine on
>> SSLCertificateFile /etc/httpd/conf/server.crt
>> SSLCertificateKeyFile /etc/httpd/conf/server.key
>> SSLProxyEngine On
>> SSLCACertificateFile/etc/httpd/conf/gsoc.pem
>> SSLProtocol all -SSLv2
>> SSLVerifyClient require
>> SSLVerifyDepth 1
>> SSLOptions +StrictRequire
>>
>> RequestHeader set X-Remote-User %{REMOTE_USER}s
>>
>> ServerName dashboards.gsoc.co.za:443
>> ServerAdmin ad...@gsoc.co.za
>> DocumentRoot/srv/http/gdf/
>> CustomLog /var/log/httpd/gdf/access combined
>> ErrorLog/var/log/httpd/gdf/error
>> LogLeveldebug
>>
>> ProxyRequests Off
>> ProxyPreserveHost Off
>> ProxyPass /gdf https://172.20.67.2:8000/gdf
>> ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
>>
>>
>> SSLRequireSSL
>> AllowOverride none
>> AuthName "GDF"
>> AuthType Basic
>> AuthDigestProvider file
>> AuthUserFile /etc/httpd/conf/passwd
>> Require ssl-verify-client
>> Require valid-user
>> Require ssl
>> Satisfy All
>>
>>
>>
>> SSLRequireSSL
>> AuthName "GDF"
>> AuthType Basic
>> AuthDigestProvider file
>> AuthUserFile /etc/httpd/conf/passwd
>> Require ssl-verify-client
>> Require valid-user
>> Require ssl
>> Satisfy All
>>
>>
>>
>>
>
>
> --
> Take Care,
>
> David Guerra
> http://frustratedtech.com/
>
304s for sure, no doubt.
Re: [users@httpd] Mod_proxy Slow After a Week
On 24 June 2013 18:27, Tom Evans wrote:
> On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel
> wrote:
> > Hello all,
> >
> > We have:
> >
> > Apache 2.2.3
> > CentOS 5.5 x86_64
> > Splunk 5.0.2
> >
> > I only know the basics but Apache has been serving us very well with the
> > below config
> >
> > and only after about a week did pages refresh very slowly, up to a
> minute to
> > reload
> >
> > sometimes.
> >
> > I tailed both httpd and splunkd logs and saw a consistent delay of 30s
> > between proxy
> >
> > and origin server (both on same host) along with plenty of 304s,
> followed by
> > what
> >
> > appeared to be some timeout and then slowly the 200s started coming back
> in.
> >
> > A restart of httpd cleared up the issue.
> >
> > I'm lost, please help.
> >
> > Could this be a caching problem?
> >
> > Thank you.
> >
> >
> > SSLEngine on
> > SSLCertificateFile /etc/httpd/conf/server.crt
> > SSLCertificateKeyFile /etc/httpd/conf/server.key
> > SSLProxyEngine On
> > SSLCACertificateFile/etc/httpd/conf/gsoc.pem
> > SSLProtocol all -SSLv2
> > SSLVerifyClient require
> > SSLVerifyDepth 1
> > SSLOptions +StrictRequire
> >
> > RequestHeader set X-Remote-User %{REMOTE_USER}s
> >
> > ServerName dashboards.gsoc.co.za:443
> > ServerAdmin ad...@gsoc.co.za
> > DocumentRoot/srv/http/gdf/
> > CustomLog /var/log/httpd/gdf/access combined
> > ErrorLog/var/log/httpd/gdf/error
> > LogLeveldebug
> >
> > ProxyRequests Off
> > ProxyPreserveHost Off
> > ProxyPass /gdf https://172.20.67.2:8000/gdf
> > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
> >
> >
> > SSLRequireSSL
> > AllowOverride none
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> >
> >
> >
> > SSLRequireSSL
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> >
> >
>
> 30 seconds is the length of the default timeout in apache.
> Unfortunately, that timeout is used in all sorts of cases, so it does
> not tell us what is timing out.
>
> As a rank guess, I would be going for DNS timeout myself. Do you have
> HostnameLookups set to "On" or "Double", or using host names in ACLs?
>
> Cheers
>
> Tom
>
> -
> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> For additional commands, e-mail: users-h...@httpd.apache.org
>
>
We are not doing any hostname ACLs.
Forgive my limited knowledge, I did RTFM before replying to make sure I
understood
the intended config and no, we are not doing any.
"HostnameLookups Off"
How come a restart "fixes" the problem?
Thank you.
Re: [users@httpd] Mod_proxy Slow After a Week
On 26 June 2013 09:59, Thomas Eckert wrote: > This is interesting. I'm seeing similar events at one of my reverse proxy. > After fixing a DNS infrastructure problem I started seeing lots of 304s > with processing times well above 5 seconds. For header-only requests that's > a lot of time. If there is any news here please let us know. At the very > least it would help me. Will do. So far so good. Thank you.
Re: [users@httpd] Mod_proxy Slow After a Week
On 26 June 2013 12:40, Tom Evans wrote: > On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel > wrote: > > Hello all, > > > > We have: > > > > Apache 2.2.3 > > CentOS 5.5 x86_64 > > Splunk 5.0.2 > > Sorry, I didn't read your original post. Please upgrade Apache to > 2.2.24 and re-test. > Thank you, will do. I'll keep an eye and advise.
Re: [users@httpd] Mod_proxy Slow After a Week
On 24 June 2013 18:27, Tom Evans wrote:
On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel wrote:
> > Hello all,
> >
> > We have:
> >
> > Apache 2.2.3
> > CentOS 5.5 x86_64
> > Splunk 5.0.2
> >
> > I only know the basics but Apache has been serving us very well with the
> > below config
> >
> > and only after about a week did pages refresh very slowly, up to a
> minute to
> > reload
> >
> > sometimes.
> >
> > I tailed both httpd and splunkd logs and saw a consistent delay of 30s
> > between proxy
> >
> > and origin server (both on same host) along with plenty of 304s,
> followed by
> > what
> >
> > appeared to be some timeout and then slowly the 200s started coming back
> in.
> >
> > A restart of httpd cleared up the issue.
> >
> > I'm lost, please help.
> >
> > Could this be a caching problem?
> >
> > Thank you.
> >
> >
> > SSLEngine on
> > SSLCertificateFile /etc/httpd/conf/server.crt
> > SSLCertificateKeyFile /etc/httpd/conf/server.key
> > SSLProxyEngine On
> > SSLCACertificateFile/etc/httpd/conf/gsoc.pem
> > SSLProtocol all -SSLv2
> > SSLVerifyClient require
> > SSLVerifyDepth 1
> > SSLOptions +StrictRequire
> >
> > RequestHeader set X-Remote-User %{REMOTE_USER}s
> >
> > ServerName dashboards.gsoc.co.za:443
> > ServerAdmin ad...@gsoc.co.za
> > DocumentRoot/srv/http/gdf/
> > CustomLog /var/log/httpd/gdf/access combined
> > ErrorLog/var/log/httpd/gdf/error
> > LogLeveldebug
> >
> > ProxyRequests Off
> > ProxyPreserveHost Off
> > ProxyPass /gdf https://172.20.67.2:8000/gdf
> > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
> >
> >
> > SSLRequireSSL
> > AllowOverride none
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> >
> >
> >
> > SSLRequireSSL
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> >
> >
>
> 30 seconds is the length of the default timeout in apache.
> Unfortunately, that timeout is used in all sorts of cases, so it does
> not tell us what is timing out.
>
> As a rank guess, I would be going for DNS timeout myself. Do you have
> HostnameLookups set to "On" or "Double", or using host names in ACLs
>Sorry, I didn't read your original post. Please upgrade Apache to
>2.2.24 and re-test.
We upgraded Apache and installed CentOS-6.4.
Problem gone.
