On Feb 25, 2014, at 10:50 AM, Charles Richard <charle...@thelearningbar.com> wrote:
> On Tue, Feb 25, 2014 at 11:37 AM, Daniel Mikusa <dmik...@gopivotal.com>wrote: > >> On Feb 25, 2014, at 10:14 AM, Charles Richard < >> charle...@thelearningbar.com> wrote: >> >>> Hi, >>> >>> On Tue, Feb 25, 2014 at 1:26 AM, Christopher Schultz < >>> ch...@christopherschultz.net> wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA256 >>>> >>>> Charles, >>>> >>>> On 2/24/14, 10:15 AM, Charles Richard wrote: >>>>> Sorry if this is not the right forum for this kind of inquiry. I >>>>> figure the best candidates would be in this forum from personal >>>>> experience. >>>>> >>>>> Our company is having production issues which I believe are either >>>>> due to application inefficiencies or a bug somewhere in our >>>>> software stack. >>>>> >>>>> We are having production issues with our Tomcat connection pool >>>>> using c3p0 and while my knowledge in this area has improved, I lack >>>>> the Java developer background that might help in this area and we >>>>> are at a point where we need this solved quickly. >>>> >>>> I've never gotten the sense that c3p0 was production-ready. What made >>>> you deploy with c2p0 instead of either of the two connection pools >>>> that ship with Tomcat? (Note that c3p0 has nothing to do with Tomcat, >>>> other than that Tomcat can be configured to use c3p0 as its >>>> connection-pool). >>>> >>>> >>> That is good to know that c3p0 might not be commonly used in production >> by >>> companies using tomcats. I was under the impression it was the most >>> commonly used. >>> >>> >>>> The problems could be related to leaked connections which I'm quite >>>>> sure we have. I have turned on c3p0 debugging and identified this >>>>> in the past and the ideal consultant could identify in our code >>>>> where those are happening and fix them. >>>> >>>> Both pools Tomcat provides can help you track-down so-called >>>> "abandoned" connections by providing stack traces that point to the >>>> line of code that obtained the connection (or even Statement or >>>> ResultSet). >>>> >>>> C3P0 also allows you to track down abandoned connections and my gut >>> feeling tells me how problem is not tied down exclusively to this as our >>> Tomcat connection pools go from 1 used connection to 150 in a minute and >> I >>> know we are not losing that many leaked connections in a minute. >>> Regardless, I will check with the manager here to see if this has been >>> tried in the past before I started this job and potentially try this as >>> unfortunately, the problem is not happening in our staging environment. >> >> When you are experiencing the problem next, take some thread dumps. Try >> to get two or three, with 10 to 15 seconds between each one. You can then >> look at the thread dumps to see what's going on here. >> >> >> https://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F >> > > It's not easy to get good thread dumps as one minute things are fine and > the next minute, the site is going downhill. I have bash scripts that > monitor c3p0 connections and take an automatic thread dump as soon as the > number exceeds what's normal. > >> >> There is information about interpreting them here or if you need help with >> it, you could post it to the list. >> >> >> https://wiki.apache.org/tomcat/HowTo#How_do_I_read_a_Java_thread_dump_.3F >> >> I have tried to analyze the thread dumps I have gotten so far without much > luck but this could just be that I don't understand enough what I'm looking > for. I will look at the link provided. I should have mentioned TDA as well. It’s as old as TDV, but works good too. https://java.net/projects/tda/downloads You can read the thread dumps in a text editor, but I find these tools help to make the process a little nicer. Dan > > Thanks for taking the time to write this! > > Dan >> >> Cheers, > Charles > >>> >>> >>>> I would highly recommend that you read this blog post I wrote several >>>> years ago that can help you look for obvious errors by providing >>>> examples for what JDBC code should look like -- if you are managing >>>> your own JDBC calls of course: >>>> >>>> >> http://blog.christopherschultz.net/index.php/2009/03/16/properly-handling-pooled-jdbc-connections/ >>>> >>>>> We are looking to hire a consultant that would come to Fredericton, >>>>> NB, Canada to work with us on this problem. Serious inquiries only. >>>>> I will be looking for proof that you have extensive experience with >>>>> Tomcat, Java Spring and c3p0. >>>>> >>>>> If you are interested, send me your resume (through your company >>>>> or individually) and send me as much proof as possible of your >>>>> experience with the specific technologies mentioned. >>>> >>>> While there is certainly no prohibition against doing so, this isn't >>>> really a "help wanted" message board. We are happy to help you -- for >>>> free! -- via email to solve your own problems. If you find this free >>>> forum helpful, please consider staying to be a part of the community. >>>> You may help others or learn things yourself. >>>> >>>> I find this forum for the most part helpful and I've posted a few times >>> before. In general though, I haven't felt that comfortable to post here. >>> >>> Before you hire anyone, you might want to try the following: >>>> >>>> 1. Swap c3p0 for something more reliable such as Tomcat's default pool >>>> (based upon commons-dbcp) or the newer, higher-performance >>>> tomcat-pool. You can find documentation for both of those online in >>>> the Users' Guide. If the pool is performing okay, this may not be >>>> necessary, but it's nearly free to try so it may be a good first-step. >>>> >>> >>> I might have to wait a month to try this so our peak traffic subsides as >> I >>> could test the config on our staging environment but since the problem is >>> only happening on production, I would be nervous of not having the >> optimal >>> configuration and make things worse. >>> >>>> >>>> 2. Check your own code for resource-handling errors. Take some time to >>>> read my blog post, or have your engineers do it (better yet, have >>>> everyone read it). >>>> >>>> Will do. >>> >>> >>>> 3. Run a static-analysis tool such as FindBugs (which is free) against >>>> your code. It can detect resource leaks you may not have known you >>>> had. Running the tool takes almost no setup and/or skill, and you can >>>> read the report entirely for free. If you have a million errors, it >>>> might take you a lot of staff to identify and fix the problems, but >>>> it's likely to point-out some problem areas -- and point you to the >>>> exact line numbers where problems are likely to be. >>>> >>>> Thanks for the recommendation, I've never heard of this tool. >>> >>> >>>> You might get a lot of mileage (kilometerage?) out of the above >>>> without having to hire anyone. >>>> >>>> Thanks for taking the time to write this and trying to help! >>> >>>> - -chris >>>> >>> >>> Cheers! >>> Charles >>> >>> >>>> -----BEGIN PGP SIGNATURE----- >>>> Version: GnuPG v1 >>>> Comment: GPGTools - http://gpgtools.org >>>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >>>> >>>> iQIcBAEBCAAGBQJTDCmHAAoJEBzwKT+lPKRYUTAP/iNhYjUQqaYIXCEn7WrKEpVW >>>> PCn4+ZLmzlmm1r7GSVbSc0+632JKA+YSOz+c8LplGnMAmUa6eU7sz2KgrunsnCCm >>>> y5UiNySJ3nQEtKcHn0c7TpNmfGUGn61Y0GoK9A+JID8jubxBNMmt4nvH7FBfRrGA >>>> GjB93hwIsJ0pK0InmejXmcoBxRYIzaVavV7nW7I/63SURv+TnXmuriHtrC5KBB39 >>>> /o5V9VNaDZqsm7zsi3qqEYoxY2/yEMt+W9++bYajXFhhhUevWdRw2NFgihyewRuL >>>> xc4GWV+5OIvSlsr0JtKXlJha8ooz1BVPigRd6vRQDaw2DFNzXy0Jl30kGK5ECuwI >>>> SmldUzYjZZGEJBhN2RGNsJgDi0zpvOIBmAciUCWZFNy7xzhubx+DTK1AHUYjW15j >>>> H1ujwZAkXfSsfY9h+vP9ATntn2YBQ3R3L7pKtjd+d7T5C9ULDIglH4o3+2kK4TEg >>>> zV+bUhyLXM/sf9IXMOTUZa9TsF8EovbaT5sRx2gXWqbMKOgyy3lJuzMaOg3110Cd >>>> P5tOfypeevWE0NY5+nUXDtmgKz+GHAkaOkdhlvDBjQ0UwNL/yV9WU5ui1pev2KLi >>>> Ct7DHj92K//ROM6RerkMjHPCcLPX5c6bbBzXl5R/unfprkPoRZ0RwgxQK/2ARAst >>>> ZDSf1VD+AUL82qVQy1rf >>>> =VgOW >>>> -----END PGP SIGNATURE----- >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
