RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
cool. that works for me. so, let's say i want to watch my sql server connections monitor and run this pool monitor from a jsp at the same time. shouldn't i expect to see the # of active + # of idle on the jsp add up to the # of connections that sql server knows about? (assuming tomcat is the only user of connections to this db) trying to check my assumptions here. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:00 PM To: [EMAIL PROTECTED] Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original Message- From: Lawrence, Gabriel [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto
Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
now, given that i can confirm that i am closing connections (calling close() on them, which returns them to the pool), can you suggest how i might locate where those connections are not getting released? i have read some stuff about dbcp not being entirely reliable in releasing connections. is that true? argh. One question. Why should DBCP *release* a connection? The way I see it, DBCP is running fine. Let me explain. The job of any JDBC connection pool is to open connections for you, to keep a pool of pre-opened connection ready to serve and to manage it. Nobody ever mentiones *closing* the connections. Well, they can be closed, usually in two situations: 1. the number of unused connections is over the maxUnused 2. the pool has decided to recycle the connections (to clean up memory leaks and garbage) You are definitely not falling under (1) - heavy load of your site will consumes more and more of the DB. You are most likely not a candidate for (2), either - after 5 minutes no reasonable connection pool will consider any particular connection stale or a candidate for recycling to prevent memory leaks. So, again, why should you see connections being closed, if the load of oyur site is growing? OTOH, if you are using connections from the pool and returning them on regular basis, then there shouldn't be an increase of numebr of open connections. If you can verify this scenario, then you have located a bug: 1. count open connections 2. open a connection from the pool 3. do something 4. close a connection (this should return it to the pool) 5. repeat a number of times (10 loops, 100 loops, 1000 loops,...) See what happens. This activity should not make any change in the number of connections. The only thing you should expect is the recycling of connections that I have mentioned, which would happen after some time or after a numebr of uses. Nix. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher. java:684) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatch er.java:575) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher .java:498) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:8 22) at org.apache.jsp._404_jsp._jspService(_404_jsp.java:373) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:247) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:193) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja va:260) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja va:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve. java:170) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:641) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:641) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java :174) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
thanks to eric for your ideas. this is a new idea but i don't know exactly how to get a thread dump. i guess it's outside the range of my experience so far. :) can you spell it out for me? much appreciated. thanks, barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:20 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Sorry, I sent this a little to soon... Get 5 or 10 thread dumps in a row real fast... just bam, bam, bam, etc... From that you can tell what the threads are doing. They should be moving through methods (or classes) very quickly. If they are sticking they are waiting on something and that's typically bad. Find what they are sticking on and fix it. :) Simple, eh? -e On Fri, 8 Aug 2003, Eric J. Pinnell wrote: Hi, You could get a thread dump and see where your connections are hanging. When you get your errors kill -QUIT the pid. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: ok, well, at least we have concrete proof of it. now, given that i can confirm that i am closing connections (calling close() on them, which returns them to the pool), can you suggest how i might locate where those connections are not getting released? i have read some stuff about dbcp not being entirely reliable in releasing connections. is that true? argh. thanks again, barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Yup, and when nothing is happening, the # of active connections should be 0. If it isn't you aren't releasing connections properly. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:19 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions cool. that works for me. so, let's say i want to watch my sql server connections monitor and run this pool monitor from a jsp at the same time. shouldn't i expect to see the # of active + # of idle on the jsp add up to the # of connections that sql server knows about? (assuming tomcat is the only user of connections to this db) trying to check my assumptions here. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:00 PM To: [EMAIL PROTECTED] Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
You might be running into this: http://www.mail-archive.com/[EMAIL PROTECTED]/msg99184.html http://www.mail-archive.com/[EMAIL PROTECTED]/msg99331.html -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:53 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions You might want to switch over to the commons-dev list at this point and talk to the dbcp people themselves. But for starters I suggest you try to build the dbcp tip and see how that goes. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 5:49 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ok, well, at least we have concrete proof of it. now, given that i can confirm that i am closing connections (calling close() on them, which returns them to the pool), can you suggest how i might locate where those connections are not getting released? i have read some stuff about dbcp not being entirely reliable in releasing connections. is that true? argh. thanks again, barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Yup, and when nothing is happening, the # of active connections should be 0. If it isn't you aren't releasing connections properly. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:19 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions cool. that works for me. so, let's say i want to watch my sql server connections monitor and run this pool monitor from a jsp at the same time. shouldn't i expect to see the # of active + # of idle on the jsp add up to the # of connections that sql server knows about? (assuming tomcat is the only user of connections to this db) trying to check my assumptions here. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:00 PM To: [EMAIL PROTECTED] Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original Message- From: Lawrence, Gabriel [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:43 AM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:38 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions I'm pretty sure can't create socket means it can't connect to the DB for some reason. I would wager the DB (or something) is refusing the connection. The fact that it said can't create socket means that tomcat is trying to do so. So that part of it seems alright. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely huge connection limit of 99,999 we think. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Is it also unlimited (or insanely huge) on the database? -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObject Pool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSourc e.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.ja va:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
You might want to switch over to the commons-dev list at this point and talk to the dbcp people themselves. But for starters I suggest you try to build the dbcp tip and see how that goes. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 5:49 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ok, well, at least we have concrete proof of it. now, given that i can confirm that i am closing connections (calling close() on them, which returns them to the pool), can you suggest how i might locate where those connections are not getting released? i have read some stuff about dbcp not being entirely reliable in releasing connections. is that true? argh. thanks again, barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Yup, and when nothing is happening, the # of active connections should be 0. If it isn't you aren't releasing connections properly. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:19 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions cool. that works for me. so, let's say i want to watch my sql server connections monitor and run this pool monitor from a jsp at the same time. shouldn't i expect to see the # of active + # of idle on the jsp add up to the # of connections that sql server knows about? (assuming tomcat is the only user of connections to this db) trying to check my assumptions here. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:00 PM To: [EMAIL PROTECTED] Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original Message- From: Lawrence, Gabriel [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:43 AM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:38 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions I'm pretty sure can't create socket means it can't connect to the DB for some reason. I would wager the DB (or something) is refusing the connection. The fact that it said can't create socket means that tomcat is trying to do so. So that part of it seems alright. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely huge connection limit of 99,999 we think. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Is it also unlimited (or insanely huge) on the database? -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(Aband onedObject Pool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(Poolin gDataSourc e.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDat aSource.ja va:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original Message- From: Lawrence, Gabriel [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:43 AM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:38 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions I'm pretty sure can't create socket means it can't connect to the DB for some reason. I would wager the DB (or something) is refusing the connection. The fact that it said can't create socket means that tomcat is trying to do so. So that part of it seems alright. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely huge connection limit of 99,999 we think. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Is it also unlimited (or insanely huge) on the database? -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what
Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher. java:684) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatch er.java:575) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher .java:498) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:8 22) at org.apache.jsp._404_jsp._jspService(_404_jsp.java:373) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:247) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:193) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja va:260) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja va:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve. java:170) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:641) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:641) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java :174) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:458) at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551) at java.lang.Thread.run(Thread.java:536) org.apache.commons.dbcp.DbcpException: java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConne ctionFactory.java:85) at
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original Message- From: Lawrence, Gabriel [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:43 AM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original Message- From: Lawrence, Gabriel [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:43 AM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:38 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions I'm pretty sure can't create socket means it can't connect to the DB for some reason. I would wager the DB (or something) is refusing the connection. The fact that it said can't create socket means that tomcat is trying to do so. So that part of it seems alright. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
ok, well, at least we have concrete proof of it. now, given that i can confirm that i am closing connections (calling close() on them, which returns them to the pool), can you suggest how i might locate where those connections are not getting released? i have read some stuff about dbcp not being entirely reliable in releasing connections. is that true? argh. thanks again, barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Yup, and when nothing is happening, the # of active connections should be 0. If it isn't you aren't releasing connections properly. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:19 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions cool. that works for me. so, let's say i want to watch my sql server connections monitor and run this pool monitor from a jsp at the same time. shouldn't i expect to see the # of active + # of idle on the jsp add up to the # of connections that sql server knows about? (assuming tomcat is the only user of connections to this db) trying to check my assumptions here. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:00 PM To: [EMAIL PROTECTED] Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
Try running netstat on the linux box to see how many connections are out there. This might give you some more visibility into what network connections are actually being used and where they are going. -gabe -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 10:43 AM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:38 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions I'm pretty sure can't create socket means it can't connect to the DB for some reason. I would wager the DB (or something) is refusing the connection. The fact that it said can't create socket means that tomcat is trying to do so. So that part of it seems alright. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely huge connection limit of 99,999 we think. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Is it also unlimited (or insanely huge) on the database? -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObject Pool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSourc e.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.ja va:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc her. java:684) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDisp atch er.java:575) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispat cher .java:498) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.ja va:8 22) at org.apache.jsp._404_jsp._jspService(_404_jsp.java:373) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely huge connection limit of 99,999 we think. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Is it also unlimited (or insanely huge) on the database? -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher. java:684) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatch er.java:575) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher .java:498) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:8 22) at org.apache.jsp._404_jsp._jspService(_404_jsp.java:373) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:247) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:193) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja va:260) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja va:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180 ) at org.apache.catalina.core.StandardPipeline
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
Yup, and when nothing is happening, the # of active connections should be 0. If it isn't you aren't releasing connections properly. -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:19 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions cool. that works for me. so, let's say i want to watch my sql server connections monitor and run this pool monitor from a jsp at the same time. shouldn't i expect to see the # of active + # of idle on the jsp add up to the # of connections that sql server knows about? (assuming tomcat is the only user of connections to this db) trying to check my assumptions here. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 4:00 PM To: [EMAIL PROTECTED] Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Oh, here is the code that will work with a JNDI resource: javax.naming.Context initContext; try { initContext = new InitialContext(); javax.naming.Context envContext = (javax.naming.Context)initContext.lookup(java:/comp/env); BasicDataSource ds = (BasicDataSource)envContext.lookup(jdbc/SessionDB); if(ds!=null){ pageContext.getOut().println( jdbc/SessionDB: Active: + ds.getNumActive() + Idle: + ds.getNumIdle() + br); } } catch (NamingException e) { // TODO Auto-generated catch block //e.printStackTrace(); } -Original Message- From: Angus Mezick Sent: Friday, August 08, 2003 3:26 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Here is a little snippet I use inside of a custom tag to disply pool health: private void printPoolStatus() throws IOException { PoolingDriver driver = new PoolingDriver(); pageContext.getOut().println(brbDbcp Pool Data:/b br); ObjectPool gop = driver.getPool(SessionManager); if (gop != null) { pageContext.getOut().println( SessionManager: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } gop = driver.getPool(CommerceServer); if (gop != null) { pageContext.getOut().println( CommerceServer: Active: + gop.getNumActive() + Idle: + gop.getNumIdle() + br); } } You WILL need to change the getPool names. --Angus -Original Message- From: Barclay A. Dunn [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:47 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions ah! there's the rub. i am also searching the archives of the jakarta-commons-users list for insight and must also post to that list if i can't figure it out myself, but we think no, they are not getting closed correctly. i can claim 100% of the time i am calling conn_.close(), but when we watch our db connections they go up but they never come down. so we think no. as i mentioned before, i do not fully grok the workings of dbcp and am thus in over my head. i could use a little step-by-step on how to test this (other than the logical deduction my sysadmin and i are using right now). thanks a million everybody for trying to be so helpful. barclay -Original Message- From: Angus Mezick [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:23 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions All of these connections are being correctly closed, right? --Angus -Original Message- From: rob engstrom [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 2:08 PM To: 'Tomcat Users List' Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Ok, I'm the sysamdin of which is being spoken. So I checked netstat, as I normally do. Here's a bit of info: netstat -an | grep 172.16 | grep ESTABLISHED | wc -l (172.16 refers to the DB connection on the internal network, on a separate VLAN than the rest of the network traffic) This elicits a response of : 45 , which if I watch the sysmon on the SQL server, it shows 45 connections currently. Same command with TIME_WAIT instead, shows: 15 So at least right now, it doesn't seem to be so much a connection problem to the DB. We did his some too many open files errors in the Catalina.out, which I temporarily resolved by changing ulimit -n from 1024 to 8092. --- robert engstrom -Original
RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions
your logic seems on target to me. my sysadmin (who's just as stymied as i am) says that according to his sources, sockets on linux are open files so if the open file limit is exceeded, it would affect sockets as well. any thoughts on this? barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:38 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions I'm pretty sure can't create socket means it can't connect to the DB for some reason. I would wager the DB (or something) is refusing the connection. The fact that it said can't create socket means that tomcat is trying to do so. So that part of it seems alright. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: well, i followed the dbcp comment that says set to 0 for unlimited and that turns out to be wrong. i also tried -1 and that is equally unuseful in terms of setting it to unlimited. so i tried setting maxActive to 10 and so far it is generating no errors. no, i'm wrong. connecs shot up to around 600 or maybe more, then we started getting the java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket. the db has an insanely huge connection limit of 99,999 we think. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:27 PM To: Tomcat Users List Subject: RE: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Is it also unlimited (or insanely huge) on the database? -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: i adjusted the pool to unlimited and am still getting these errors. something else seems to be fekachte. barclay -Original Message- From: Eric J. Pinnell [mailto:[EMAIL PROTECTED] Sent: Friday, August 08, 2003 1:11 PM To: Tomcat Users List Subject: Re: Help! heavy traffic is crapping out our site every 5 min! DBCP exceptions Hi, Your pool of connections to your database is exhausted. Try upping the number. -e On Fri, 8 Aug 2003, Barclay A. Dunn wrote: we are getting a ton of these errors in our catalina.out and i could use help in fixing it. i know they are related to our connection pooling, but not what to do to fix. my understanding of the underlying mechanics of connection pooling is somewhat limited. i have put two different but related error messages in here: java.sql.SQLException: DBCP could not obtain an idle db connection, pool exhausted at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool .java:123) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:3 12) at com.happypuppy.util.HPSql.getConnection(HPSql.java:21) at org.apache.jsp.index_jsp._jspService(index_jsp.java:222) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher. java:684) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatch er.java:575) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher .java:498) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:8 22) at org.apache.jsp._404_jsp._jspService(_404_jsp.java:373) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:2 04) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:247) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:193) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja va:260