Well, it appears that we are slowly getting to the bottom of this. But
with every answer, I get a few more questions....
First, I installed the latest TC8 on my laptop, copied my server.conf
and conf/catalina folder to it and started it up just to see what errors
I got. After changing out an obsolete listener, it came up. I found
all of the <resource> parm exceptions in stderr. So that question is
cleared up. Thanks for clarifying where to find that.
The site is a wedding vendor advertiser site that spans two major
cities. There is no user login. Simply a very huge online catalog.
I'm certain it's deployed only once. Whether I need that many
connections is a valid question. As far as I know, I haven't hit the
limit in normal operation until now. Could possibly reduce the count if
I collect statistics.
I've been monitoring the production server logs all day watching to be
sure connection pool doesn't dry up again. About an hour ago, there was
a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
They showed connections from 1am right after my last bounce of the
server thru 1:35pm. The good news is with the stack trace on one of
them I was able to see the bug causing the leak. But why did it decide
to wait over 12 hours accumulating abandoned connections before dumping
them back in the pool? I realize from now knowing the code bug that the
leak is a slow drip that is continually leaking on a regular basis. But
since that last 12-hour accumulated dump, the abandoning has returned to
silence. Since leaks are occurring regularly and would be timing out
regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
in stdout instead of a big dump every 12 hours?
It's going to take a day or two to fix the leak, test, and deploy. If
indeed abandoned connections are now correctly being returned to the
pool, then I presume we are back to working the way it did on TC7.
Still not sure why it started working now. But I guess once I get the
leak fixed and if TC8 is now configured to handle abandoned connections,
I'm good. Still would like to know about the mega-dump vs. trickle of
abandoned connections being logged.
Thanks again for all of the info.
Jerry
On 7/29/2015 1:15 PM, Christopher Schultz wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jerry,
On 7/29/15 12:06 PM, Jerry Malcolm wrote:
On 7/29/2015 10:11 AM, Christopher Schultz wrote:
I guess I could move up to the latest 8. But it'll require some
client down time, and they aren't thrilled with that at this
point.
Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?
Off topic, but relevant.... it sure would be nice if the xml
parser for <resource> tags and other config items would flag
with errors any parameter misspellings such as using the old
obsolete 'removeAbandoned' instead of the new
'removeAbandonedOnBorrow', etc. or even just typo misspells.
It does:
Jul 02, 2015 10:17:25 AM
org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
ObjectInstance WARNING: Name = diagnosis Property maxActive is
not used in DBCP2, use maxTotal instead. maxTotal default value
is 8. You have set value of "1" for "maxActive" property, which
is being ignored.
...
WARNING: Name = diagnosis Property removeAbandoned is not used
in DBCP2, use one or both of removeAbandonedOnBorrow or
removeAbandonedOnMaintenance instead. Both have default value set
to false. You have set value of "true" for "removeAbandoned"
property, which is being ignored.
(I have a context.xml that I want to use between both Tomcat 7
and Tomcat 8, so I actually have both spellings of those
attributes in my XML.)
This is really strange. Are the above log entries in stdout?
Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.
I had 'removeAbandoned' in my resource statements until yesterday.
But I am not seeing anything in stdout like the statement above.
And after changing removeAbandoned to the correct spelling, I'm
also not seeing any abandoned log entries either.
Hmm.
Is there some sort of 'log level' that may be throttling back my
log entries? If my code has leaks, and I'm running out of
connections but not seeing any abandoned entries in the log, that
should be a concern, right?
Possibly.
Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.
What does your <Resource> element actually look like? (Remember
to remove sensitive information such as passwords.)
<Resource name="jdbc/wgbe" testOnBorrow="true"
validationQuery="SELECT 1"
validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.
Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.
auth="Container" type="javax.sql.DataSource" maxTotal="300"
That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?
maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
driverClassName="org.gjt.mm.mysql.Driver"
url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
Do you have the 'manager' web application installed? Can you check to
see what applications are actually deployed? If you have 2 copies
deployed, you might see 600 connections open.
Also, do you have any stray context.xml files hanging around in
conf/*/* that you weren't expecting? If you are making changes to your
configuration and they don't appear to be having any effect, it's
possible that your new configuration isn't being applied because
another config is masking it.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVuRgxAAoJEBzwKT+lPKRY/2YQALiweWD9Le3Ip3LeyFL/c43d
ngd50y3iqYcPGbA33IOAy5ThDmcrRDiSbs5cMG4hhAo3rFIU8QmrCJdSFrhLIXu4
K0ZcZPtlmPrEL0CUUE2D6HIX4mg0FMiNB7c0N4XjB6iicd16VGEMBVs4Ck7Sr0DM
A2zTFEYhH/YXlWoOIkTE8gduNfHNNM7y6aaspIgRy9gqAMRFvD4HZaXONCrXPyKD
hh2o8dInaSZY/jv3HrGcZd0SG9LzggKUynN195SOhc9cy6tsvOeIRFiOZWKckYSz
NpKg77+2q9SYFVM5GSKPdrFlzbcjavmsoAP3EvCtWV5HvFE7MHo/ajvI/rSb47Uy
3jYD2b7S9NZdvPu/d8EPlqL29gbUhK2a1ISIY0S3Dgyi/Junzps/sztmOfgib71X
IRDa3KH6gww4IwOX9K8YmMfHi9On06M2NF9NOVYiLqEN4CkHPAQ8YhzpnQz7/LSg
DsxLQ9D5lDdaBjE3kvL17kd9h969SgM2SJKHfHay2xOTYjOHQ9vHY1EjF/C+3smd
BzChYmUmjy0CBWjSCfYxJTjShNkLvyRyenAE+5WAFZhpOGTGW2DxSCXf9zQ2rqlv
vSk7xVf6AMrZXQEZ376MsbRXGlUipZ9ZlufXLj9Pdqmcad7WoLHGaKNNwIyJvdWy
AVhayMcrSTj5Y/xBeSXQ
=xsp7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
