Re: Failing Mysql JDBC connections
I wrote: As soon as this happens, all pages being worked on at the same time fail with such messages as: SQLTransformer.Query: Failed to execute query [...] com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: No operations allowed after connection closed. For the record, this was caused by a deadlock problem, where many simultaneous requests would get a connection for their SQLTransformer, exhausting the connection pool, and then get stuck requesting more connections through an input module I wrote myself, before the SQLTransformers had a chance to return the connections to the pool. I worked around the problem by uncapping the database connections: pool-controller min=100 max=150 max-strict=false/ Now I'd like to know how to avoid the deadlock condition in the first place, with capped connections, but I'm not sure it can be done. I've asked the dev list. Tobia - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Failing Mysql JDBC connections
2008/6/20 Tobia Conforto [EMAIL PROTECTED]: For the record, this was caused by a deadlock problem, where many simultaneous requests would get a connection for their SQLTransformer, exhausting the connection pool, and then get stuck requesting more connections through an input module I wrote myself, before the SQLTransformers had a chance to return the connections to the pool. I worked around the problem by uncapping the database connections: pool-controller min=100 max=150 max-strict=false/ Now I'd like to know how to avoid the deadlock condition in the first place, with capped connections, but I'm not sure it can be done. I've asked the dev list. Instead of max-strict=false have you tried blocking=false? I believe that'll cause it to throw an exception when the pool runs out rather than waiting till another one comes available (which is the reason it deadlocks). Although you probably don't want the users to get a system error page, it's better than the whole app locking up. Also try adding a timeout value (e.g. timeout=500), which I think means that even if it's blocking, it won't do so indefinitely. Not that you'll necessarily notice, as the SQLTransformer tries again a number of times after a delay (both configurable), and by default it'll be 30 seconds or more before it finally gives up. We had a similar problem, just using multiple SQLTransformer components (no input module in our case) in a single pipeline. We fixed it by changing the transformer so that it returned its connection to the pool sooner (in the endDocument method, if I remember correctly, or maybe it was the endElement for the top-level execute-query element) instead of the recycle method. The components in a pipeline are all created (or got from their respective pools) when the pipeline is being built, before it executes, and it only releases/recycles them all once the pipeline finishes executing, so the existing code meant that one pipeline would grab multiple database connections from the pool but not return any of them until the whole thing was done. This meant a handful of simultaneous calls to a pipeline(s) with a few SQLTransformers could deadlock each other (actually, under a high enough load just having two SQLTransformers in a pipeline would be enough) as the transformers later on in the pipeline all wait for a connection (stopping the pipelines from completing) that isn't going to become available as the earlier transformers are still keeping hold of them all... By closing the connections as soon as they're finished with, they become available again for the later transformers to use. I can't remember if I raised a bug/enhancement request in JIRA to change the transformer's behaviour in this way; if I didn't, perhaps you'd like to... Andy. -- http://pseudoq.sourceforge.net/ Open source java Sudoku solver - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Failing Mysql JDBC connections
Hello I have a big problem right now: my Cocoon MySql connections have started to fail randomly, and when they do they bring down all pages currently being generated. My gut feeling is that it's a synchronization problem, some race condition in Cocoon or in the Mysql connector. This is because the failures happen on random pages, that usually render correctly, and also because the higher the concurrency of http requests, the higher the rate of failure. Apparently I didn't stress-test the server configuration with a high enough concurrency, before going online. Here is what I see in the logs: SQLTransformer.Query: Failed to execute query [...] com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com .mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java: 113) ... (other mysql jdbc classes follow, then excalibur, then the SQLTransformer) Looking at a network dump, I can see a series of Mysl protocol QUIT messages that tear down all tcp connections to the Mysql server. It happens simultaneously on ALL tcp connections and I can see the Jdbc connector sending appropriate QUIT messages to the server, so it's not a hard connection reset, coming from the network stack. The Mysql server is on localhost, so it's not a network problem. As soon as this happens, all pages being worked on at the same time fail with such messages as: SQLTransformer.Query: Failed to execute query [...] com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: No operations allowed after connection closed. or: ErrorHandlerHelper: Failed to process pipeline java.lang.NullPointerException at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:966) ... or other such NPE inside the JDBC driver, on lines where the driver is trying to use its own iternal connection object (which now appears to be null.) After that, the database connections are reopened and Cocoon proceeds normally, but after a few of these error iterations it all hangs. (Memory leak? I don't know.) I'm using Cocoon 2.1.10 and 2.1.11 (same results) and MySql connector 3.1.14 and 5.0.6 (same results.) I'm using SQLTransformer with prepared statements (sql:in-parameter) and a few custom actions of mine which use JDBI prepared statements. The server runs Debian stable with its latest Sun Java 1.5.0 and Mysql 5.0.32 packages. I cannot use the 5.0.8 or 5.1.6 connectors because they don't work with Cocoon 2.1 (a strange bug with column aliases, aka. SELECT column AS alias, worthy of another thread.) Has anybody seen anything like this before? Do you want me to try and build a minimal example that shows the error? Could it be the prepared statements' fault? Any help will be appreciated. Tobia - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
