Re: major deadlock issues (JDBC store with MySQL)
So, you say you have good results with the tx file stores? And even good ones with caching turned on and MySQL? Hmmm. I had a second look at the exceptions and the original message is that a lock has timed out, not that there is a deadlock. This is what the JDBC driver seems to add. Now, one thing I know for sure is when load on the db server is very high connections / transactions can time out resulting in these sorts of errors which are displayed in your traces. Same thing when you try to acquire a connection from your db server when there are no more available. That's why it is important to have at most the number of connections in your pool you have configured in your db. Maybe the bug I fixed was not related to your problem at all, but just caused by too high load? One thing I can recommand is to lower the isolation level to READ_COMMITTED as the WebDAV layer does very restrictive concurrency checking. Concerning the testsuite: It will need parameters. Try running something like functional.bat or look into it to find out what is done there. Oliver On Wed, 24 Nov 2004 11:21:58 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Unfortunately I am still getting deadlocks with the latest code from the release branch. I think my initial tests were a little too aggressive, as I was generating more requests than the server could actually handle. I have now adjusted my tests to closely match the realistic throughput that I can achieve, and I'm getting good results using the tx store. The db store still isn't happy. I still get an XAException, preceded by HeuristicMixedException, IllegalStateException, all of which result in a WebdavException (Internal server error), as well as occasional SQLException and ServiceAccessException that mention deadlocks - see below. Could someone help me get the test suite running? I'll try to reproduce the problem that way, which might simplify things. I've been able to compile the test suite, but am not sure how to configure and invoke it, as I always get a success message when I run the TProcessor, even when my Slide server is down... -Mirko Exceptions: = 24 Nov 2004 11:10:57 - org.apache.slide.transaction.SlideTransaction - WARNING - Enlist failure: Resource manager [EMAIL PROTECTED] Error code XAER_NOTA in Transaction 168 xid http-8080-Processor15-1101323457477-168- in thread http-8080-Processor15 http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 51 ms, /db/load_test/user_12/foobar_7.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 39 ms, /db/load_test/user_12/foobar_6.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 42 ms, /db/load_test/user_12/foobar_5.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 65 ms, /db/load_test/user_12/foobar_4.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 56 ms, /db/load_test/user_12/foobar_3.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 163 ms, /db/load_test/user_12/foobar_2.txt 24 Nov 2004 11:10:58 - org.apache.slide.transaction.SlideTransaction - WARNING - Commit failure: Resource manager [EMAIL PROTECTED] Error code XAER_NOTA in Transaction 168 xid http-8080-Processor15-1101323457477-168- in thread http-8080-Processor15 javax.transaction.xa.XAException at org.apache.commons.transaction.util.xa.AbstractXAResource.commit(AbstractXAResource.java:69) at org.apache.slide.transaction.SlideTransaction.commit(SlideTransaction.java:300) at org.apache.slide.transaction.SlideTransactionManager.commit(SlideTransactionManager.java:186) at org.apache.slide.common.NamespaceAccessTokenImpl.commit(NamespaceAccessTokenImpl.java:423) at org.apache.slide.webdav.method.AbstractWebdavMethod.run(AbstractWebdavMethod.java:388) at org.apache.slide.webdav.WebdavServlet.service(WebdavServlet.java:155) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.slide.webdav.filter.LogFilter.doFilter(LogFilter.java:141) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
RE: major deadlock issues (JDBC store with MySQL)
Great work Oliver! I'll grab that fix too. You can never have too many deadlock fixes, I always say ;-) Warwick -Original Message- From: Oliver Zeigermann [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 24, 2004 5:46 AM To: Slide Users Mailing List Subject: Re: major deadlock issues (JDBC store with MySQL) Inside security code I have found an additional request which was done outside of the transaction which explains why the locking in the WebDAV layer failed: different transactions inside the same requests where blocking each other. Now everything should be done in one and a single request. The new testcase passes now, Mirko, would you do us all the favor and check out the latest sources either from release or head branch and see if your test case succeeds as well? If not please supply stack traces and hints how to use your JMeter tests. Thanks, Oliver On Wed, 24 Nov 2004 00:53:01 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: I just could reproduce the problem with the new test case testsuite/junit/xmltestcases/functional/extra/multi-user/getPut/getPut Folder.xml and it also exists in the latest sources. It has got something to do with security and seems to be what Warwick already identified in http://issues.apache.org/bugzilla/show_bug.cgi?id=31907 which we thought was solved, but apparently is not. I will have a look at this ASAP. Oliver On Fri, 19 Nov 2004 15:59:22 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I did some basic JMeter concurrency testing to validate if Slide meets our concurrency requirements. Unfortunately, the results were not very promising. As I mentioned, we are planning on using Slide as a repository for application data. We expect to have a few hundred concurrent users of our application who would be reading from as well as (although less frequently) writing to the repository. In case it matters, most of the read and write access would be to a dedicated folder per user, rather than a shared folder. I realize that this use case is probably very different from the way most people use Slide, which I assume is mostly for web publishing. My test mainly consists of a JSP page that uses the WebDAV client library to access a folder, iterate over the 10 contained documents and retrieve their contents (about 2k each), and create a new document, retrieve it, and then delete it. I have 100 different folders, all populated with the same 10 documents. My JMeter test consists of 5 concurrent threads that each access a different one of the 100 folders, with 800ms between requests. After only a few seconds, Slide completely deadlocks with various exceptions to that effect in the Tomcat logs. I have tried increasing my JDBC store's MySQL connection pool from 10 to 30, without any improvements. It seems like creating documents is responsible for the deadlocks, as the test behaved fine when it was limited to read-only access. Can anyone think of a way for me to tune the system in order to prevent the deadlocks from occurring? -Mirko - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: major deadlock issues (JDBC store with MySQL)
This is probably no surprise, but once I enable caching (as we would in production), everything works fine. My test case consists of about 90% read-only access and 10% read/write access, which should reflect our expected usage patterns. -Mirko On Wed, 2004-11-24 at 11:21, Mirko Froehlich wrote: Unfortunately I am still getting deadlocks with the latest code from the release branch. I think my initial tests were a little too aggressive, as I was generating more requests than the server could actually handle. I have now adjusted my tests to closely match the realistic throughput that I can achieve, and I'm getting good results using the tx store. The db store still isn't happy. I still get an XAException, preceded by HeuristicMixedException, IllegalStateException, all of which result in a WebdavException (Internal server error), as well as occasional SQLException and ServiceAccessException that mention deadlocks - see below. Could someone help me get the test suite running? I'll try to reproduce the problem that way, which might simplify things. I've been able to compile the test suite, but am not sure how to configure and invoke it, as I always get a success message when I run the TProcessor, even when my Slide server is down... -Mirko Exceptions: = 24 Nov 2004 11:10:57 - org.apache.slide.transaction.SlideTransaction - WARNING - Enlist failure: Resource manager [EMAIL PROTECTED] Error code XAER_NOTA in Transaction 168 xid http-8080-Processor15-1101323457477-168- in thread http-8080-Processor15 http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 51 ms, /db/load_test/user_12/foobar_7.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 39 ms, /db/load_test/user_12/foobar_6.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 42 ms, /db/load_test/user_12/foobar_5.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 65 ms, /db/load_test/user_12/foobar_4.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 56 ms, /db/load_test/user_12/foobar_3.txt http-8080-Processor16, 24-Nov-2004 11:10:57, unauthenticated, GET, 200 OK, 163 ms, /db/load_test/user_12/foobar_2.txt 24 Nov 2004 11:10:58 - org.apache.slide.transaction.SlideTransaction - WARNING - Commit failure: Resource manager [EMAIL PROTECTED] Error code XAER_NOTA in Transaction 168 xid http-8080-Processor15-1101323457477-168- in thread http-8080-Processor15 javax.transaction.xa.XAException at org.apache.commons.transaction.util.xa.AbstractXAResource.commit(AbstractXAResource.java:69) at org.apache.slide.transaction.SlideTransaction.commit(SlideTransaction.java:300) at org.apache.slide.transaction.SlideTransactionManager.commit(SlideTransactionManager.java:186) at org.apache.slide.common.NamespaceAccessTokenImpl.commit(NamespaceAccessTokenImpl.java:423) at org.apache.slide.webdav.method.AbstractWebdavMethod.run(AbstractWebdavMethod.java:388) at org.apache.slide.webdav.WebdavServlet.service(WebdavServlet.java:155) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.slide.webdav.filter.LogFilter.doFilter(LogFilter.java:141) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at
Re: major deadlock issues (JDBC store with MySQL)
I just tried it again with compression turned on, and the result was only marginally better. This time, the system didn't deadlock completely until after 63 iterations, whereas before it was closer to 10... The exceptions seem to look slightly different, too (I don't think I noticed a HeuristicMixedException before). They still lead to the same deadlock exception, though. See below. I have tried to attach my JMeter test case, but haven't been lucky getting the attachments getting accepted by the mailing list server so far. I'll follow up in a separate email. Unfortunately I didn't have much time yesterday to look into this issue. Today, I am hoping to try the patch that Warwick mentioned, as well as run the test against the file store. I will also try your JUnit test. Thanks again for your help. -Mirko Exceptions: = 22 Nov 2004 17:26:31 - org.apache.slide.transaction.SlideTransaction - WARNING - Enlist failure: Resource manager [EMAIL PROTECTED] Error code XAER_NOTA in Transaction 1272 xid http-8080-Processor16-1101173191802-1272- in thread http-8080-Processor16 http-8080-Processor9, 22-Nov-2004 17:26:31, unauthenticated, GET, 200 OK, 225 ms, /db/load_test/user_6/dummy_1101173191200 http-8080-Processor15, 22-Nov-2004 17:26:31, unauthenticated, GET, 200 OK, 61 ms, /db/load_test/user_8/foobar_3.txt http-8080-Processor15, 22-Nov-2004 17:26:31, unauthenticated, GET, 200 OK, 42 ms, /db/load_test/user_8/foobar_2.txt http-8080-Processor15, 22-Nov-2004 17:26:31, unauthenticated, GET, 200 OK, 42 ms, /db/load_test/user_8/foobar_1.txt 22 Nov 2004 17:26:32 - org.apache.slide.transaction.SlideTransaction - WARNING - Commit failure: Resource manager [EMAIL PROTECTED] Error code XAER_NOTA in Transaction 1272 xid http-8080-Processor16-1101173191802-1272- in thread http-8080-Processor16 javax.transaction.xa.XAException at org.apache.commons.transaction.util.xa.AbstractXAResource.commit(AbstractXAResource.java:69) at org.apache.slide.transaction.SlideTransaction.commit(SlideTransaction.java:300) at org.apache.slide.transaction.SlideTransactionManager.commit(SlideTransactionManager.java:186) at org.apache.slide.common.NamespaceAccessTokenImpl.commit(NamespaceAccessTokenImpl.java:423) at org.apache.slide.webdav.method.AbstractWebdavMethod.run(AbstractWebdavMethod.java:388) at org.apache.slide.webdav.WebdavServlet.service(WebdavServlet.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.slide.webdav.filter.LogFilter.doFilter(LogFilter.java:141) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:535) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at
Re: major deadlock issues (JDBC store with MySQL)
I have attached my JMeter test case, as well as my test JSPs. populate.jsp is used to populate the repository before running the test. It assumes that the /slide/db/load_test/ folder exists and creates 100 subfolders called user_[i] with 10 documents each. slide.jsp is the page that my JMeter test case is hitting. JMeter passes the folder name as a parameter, and the test then iterates over the documents in the folder, retrieves their content, creates a new document, retrieves its content, and deletes it. The test case uses 5 concurrent threads, and each thread pauses for 800ms between requests. Hope the attachments make it through... :) -Mirko On Sat, 2004-11-20 at 12:35, Oliver Zeigermann wrote: This should not be caused by using MySQL 4 as the Slide code for both is identical when it comes to inserting content. Additionally, serilzability of requests is handled in the WebDAV layer, so there really should be no deadlocks in the store except there are programming errors (in the store implementation). You do not use externally controlled transactions, do you? Also make sure that there are not open connections with pending transactions from other tools like e.g. an SQL command line tool or a visualizer or anything. Anyway, looking forward to your test case :) Oliver ?xml version=1.0 encoding=UTF-8? node testelement class=org.apache.jmeter.testelement.TestPlan testelement class=org.apache.jmeter.config.Arguments name=TestPlan.user_defined_variables property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.gui_classorg.apache.jmeter.config.gui.ArgumentsPanel/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.test_classorg.apache.jmeter.config.Arguments/property collection class=java.util.ArrayList propType=org.apache.jmeter.testelement.property.CollectionProperty name=Arguments.arguments/ property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.nameUser Defined Variables/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.BooleanProperty name=TestElement.enabledtrue/property /testelement property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.gui_classorg.apache.jmeter.control.gui.TestPlanGui/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.BooleanProperty name=TestPlan.serialize_threadgroupsfalse/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.test_classorg.apache.jmeter.testelement.TestPlan/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.nameSlide Test/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.BooleanProperty name=TestPlan.functional_modefalse/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.BooleanProperty name=TestElement.enabledtrue/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestPlan.comments/ /testelement node testelement class=org.apache.jmeter.threads.ThreadGroup property xml:space=preserve propType=org.apache.jmeter.testelement.property.LongProperty name=ThreadGroup.start_time1100738492000/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.test_classorg.apache.jmeter.threads.ThreadGroup/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=ThreadGroup.delay/ property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=ThreadGroup.duration/ property xml:space=preserve propType=org.apache.jmeter.testelement.property.BooleanProperty name=TestElement.enabledtrue/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=ThreadGroup.num_threads5/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.BooleanProperty name=ThreadGroup.schedulerfalse/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.gui_classorg.apache.jmeter.threads.gui.ThreadGroupGui/property testelement class=org.apache.jmeter.control.LoopController name=ThreadGroup.main_controller property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty name=TestElement.gui_classorg.apache.jmeter.control.gui.LoopControlPanel/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.IntegerProperty name=LoopController.loops-1/property property xml:space=preserve propType=org.apache.jmeter.testelement.property.StringProperty
Re: major deadlock issues (JDBC store with MySQL)
Attachments made it to the list. Would be great if you could provide a test case for our test suite as I have no idea how to let your test run. Anyone else? Oliver On Tue, 23 Nov 2004 09:21:06 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I have attached my JMeter test case, as well as my test JSPs. populate.jsp is used to populate the repository before running the test. It assumes that the /slide/db/load_test/ folder exists and creates 100 subfolders called user_[i] with 10 documents each. slide.jsp is the page that my JMeter test case is hitting. JMeter passes the folder name as a parameter, and the test then iterates over the documents in the folder, retrieves their content, creates a new document, retrieves its content, and deletes it. The test case uses 5 concurrent threads, and each thread pauses for 800ms between requests. Hope the attachments make it through... :) -Mirko On Sat, 2004-11-20 at 12:35, Oliver Zeigermann wrote: This should not be caused by using MySQL 4 as the Slide code for both is identical when it comes to inserting content. Additionally, serilzability of requests is handled in the WebDAV layer, so there really should be no deadlocks in the store except there are programming errors (in the store implementation). You do not use externally controlled transactions, do you? Also make sure that there are not open connections with pending transactions from other tools like e.g. an SQL command line tool or a visualizer or anything. Anyway, looking forward to your test case :) Oliver - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: major deadlock issues (JDBC store with MySQL)
I'll take a look at your test suite and see if I can adapt it. My test requires JMeter. In case you want to try to run it, you should be able to load the test case as it is. It hits http://localhost:8080/jmeter_slide/slide.jsp;, which is where my slide.jsp is deployed (this URL can be changed in the test case's HTTP Request node). This web app has to have all the webdavclient JAR files. I would post the WAR file, but it's too large for the mailing list. But I'll check out your test suite as well. -Mirko On Tue, 2004-11-23 at 09:59, Oliver Zeigermann wrote: Attachments made it to the list. Would be great if you could provide a test case for our test suite as I have no idea how to let your test run. Anyone else? Oliver On Tue, 23 Nov 2004 09:21:06 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I have attached my JMeter test case, as well as my test JSPs. populate.jsp is used to populate the repository before running the test. It assumes that the /slide/db/load_test/ folder exists and creates 100 subfolders called user_[i] with 10 documents each. slide.jsp is the page that my JMeter test case is hitting. JMeter passes the folder name as a parameter, and the test then iterates over the documents in the folder, retrieves their content, creates a new document, retrieves its content, and deletes it. The test case uses 5 concurrent threads, and each thread pauses for 800ms between requests. Hope the attachments make it through... :) -Mirko On Sat, 2004-11-20 at 12:35, Oliver Zeigermann wrote: This should not be caused by using MySQL 4 as the Slide code for both is identical when it comes to inserting content. Additionally, serilzability of requests is handled in the WebDAV layer, so there really should be no deadlocks in the store except there are programming errors (in the store implementation). You do not use externally controlled transactions, do you? Also make sure that there are not open connections with pending transactions from other tools like e.g. an SQL command line tool or a visualizer or anything. Anyway, looking forward to your test case :) Oliver - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: major deadlock issues (JDBC store with MySQL)
I just could reproduce the problem with the new test case testsuite/junit/xmltestcases/functional/extra/multi-user/getPut/getPutFolder.xml and it also exists in the latest sources. It has got something to do with security and seems to be what Warwick already identified in http://issues.apache.org/bugzilla/show_bug.cgi?id=31907 which we thought was solved, but apparently is not. I will have a look at this ASAP. Oliver On Fri, 19 Nov 2004 15:59:22 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I did some basic JMeter concurrency testing to validate if Slide meets our concurrency requirements. Unfortunately, the results were not very promising. As I mentioned, we are planning on using Slide as a repository for application data. We expect to have a few hundred concurrent users of our application who would be reading from as well as (although less frequently) writing to the repository. In case it matters, most of the read and write access would be to a dedicated folder per user, rather than a shared folder. I realize that this use case is probably very different from the way most people use Slide, which I assume is mostly for web publishing. My test mainly consists of a JSP page that uses the WebDAV client library to access a folder, iterate over the 10 contained documents and retrieve their contents (about 2k each), and create a new document, retrieve it, and then delete it. I have 100 different folders, all populated with the same 10 documents. My JMeter test consists of 5 concurrent threads that each access a different one of the 100 folders, with 800ms between requests. After only a few seconds, Slide completely deadlocks with various exceptions to that effect in the Tomcat logs. I have tried increasing my JDBC store's MySQL connection pool from 10 to 30, without any improvements. It seems like creating documents is responsible for the deadlocks, as the test behaved fine when it was limited to read-only access. Can anyone think of a way for me to tune the system in order to prevent the deadlocks from occurring? -Mirko - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: major deadlock issues (JDBC store with MySQL)
Actually I haven't committed the latest transaction deadlock fixes I made to get the enlistment of slide server transactions into the client's transaction working. If Mirko is using client side transactions then this may fix it. And even if he's not it still may fix it :-) But I'm still waiting on a comitter account from Apache to commit this fix. What I found particularly interesting about the exception stack that accompanies the deadlock is that it is in the thread of execution for a commit which is exactly where this latest problem was. The defect for this problem is: http://issues.apache.org/bugzilla/show_bug.cgi?id=32250 If you think the problem may be security related then this is the defect I opened for some other potentially problematic areas in the security impl that could suffer the same problem that I fixed in 31907. eg. http://issues.apache.org/bugzilla/show_bug.cgi?id=31908 Though the fact that the stack indicates its occurring during the commit would make me think it is defect 32250. Warwick -Original Message- From: Oliver Zeigermann [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 23, 2004 5:53 PM To: Slide Users Mailing List Subject: Re: major deadlock issues (JDBC store with MySQL) I just could reproduce the problem with the new test case testsuite/junit/xmltestcases/functional/extra/multi-user/getPu t/getPutFolder.xml and it also exists in the latest sources. It has got something to do with security and seems to be what Warwick already identified in http://issues.apache.org/bugzilla/show_bug.cgi?id=31907 which we thought was solved, but apparently is not. I will have a look at this ASAP. Oliver On Fri, 19 Nov 2004 15:59:22 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I did some basic JMeter concurrency testing to validate if Slide meets our concurrency requirements. Unfortunately, the results were not very promising. As I mentioned, we are planning on using Slide as a repository for application data. We expect to have a few hundred concurrent users of our application who would be reading from as well as (although less frequently) writing to the repository. In case it matters, most of the read and write access would be to a dedicated folder per user, rather than a shared folder. I realize that this use case is probably very different from the way most people use Slide, which I assume is mostly for web publishing. My test mainly consists of a JSP page that uses the WebDAV client library to access a folder, iterate over the 10 contained documents and retrieve their contents (about 2k each), and create a new document, retrieve it, and then delete it. I have 100 different folders, all populated with the same 10 documents. My JMeter test consists of 5 concurrent threads that each access a different one of the 100 folders, with 800ms between requests. After only a few seconds, Slide completely deadlocks with various exceptions to that effect in the Tomcat logs. I have tried increasing my JDBC store's MySQL connection pool from 10 to 30, without any improvements. It seems like creating documents is responsible for the deadlocks, as the test behaved fine when it was limited to read-only access. Can anyone think of a way for me to tune the system in order to prevent the deadlocks from occurring? -Mirko - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: major deadlock issues (JDBC store with MySQL)
I'm not explicitly using any transactions, unless the webdav client library transparently uses transactions. -Mirko On Tue, 2004-11-23 at 16:06, Warwick Burrows wrote: Actually I haven't committed the latest transaction deadlock fixes I made to get the enlistment of slide server transactions into the client's transaction working. If Mirko is using client side transactions then this may fix it. And even if he's not it still may fix it :-) But I'm still waiting on a comitter account from Apache to commit this fix. What I found particularly interesting about the exception stack that accompanies the deadlock is that it is in the thread of execution for a commit which is exactly where this latest problem was. The defect for this problem is: http://issues.apache.org/bugzilla/show_bug.cgi?id=32250 If you think the problem may be security related then this is the defect I opened for some other potentially problematic areas in the security impl that could suffer the same problem that I fixed in 31907. eg. http://issues.apache.org/bugzilla/show_bug.cgi?id=31908 Though the fact that the stack indicates its occurring during the commit would make me think it is defect 32250. Warwick
RE: major deadlock issues (JDBC store with MySQL)
Ok, then you aren't using client transactions as you actually need to call the WebdavResource.startTransaction() interface from somewhere in your client code for it to use client transactions. But aspects of the fix for defect 32250 may still apply. For example there were server side issues with the enlistment of DAV lock/unlock methods vs lock/unlock methods performed as part of the transaction process. Transactions are started by performing a special lockMethod call and committed or aborted with a corresponding unlockMethod call. Any time that you are enlisted in a transaction and try to make a DB store access (read or write) from the thread and its not enlisted to the same transaction then you will deadlock. Are you building the slide server/client source yourself or are you using the 2.1b2 binaries? If you're using the 2.1b2 release binaries (or earlier) then you won't even have the fix for 31907 and my suggestion would be to build the source from the latest on the 2.1 release branch yourself and try again with binaries that at least the patch for 31907. If you're already using the latest source from the 2.1 release branch then you will already have the fix for defect 31907 and you could try to take my patch files from defect 32250 and apply those to your source to see if they fix your problem. The link to the patches in bugzilla is: http://issues.apache.org/bugzilla/attachment.cgi?id=13468 You won't need the alternative patch patch, just the one at the link above. If you try this then be aware when you apply the patches to the latest CVS source that someone may have made other fixes to the CVS files since I first created the patches. I was going to integrate these fixes myself when I have committer access but you may not want to wait until that happens :-) Warwick -Original Message- From: Mirko Froehlich [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 23, 2004 6:30 PM To: Slide Users Mailing List Subject: RE: major deadlock issues (JDBC store with MySQL) I'm not explicitly using any transactions, unless the webdav client library transparently uses transactions. -Mirko On Tue, 2004-11-23 at 16:06, Warwick Burrows wrote: Actually I haven't committed the latest transaction deadlock fixes I made to get the enlistment of slide server transactions into the client's transaction working. If Mirko is using client side transactions then this may fix it. And even if he's not it still may fix it :-) But I'm still waiting on a comitter account from Apache to commit this fix. What I found particularly interesting about the exception stack that accompanies the deadlock is that it is in the thread of execution for a commit which is exactly where this latest problem was. The defect for this problem is: http://issues.apache.org/bugzilla/show_bug.cgi?id=32250 If you think the problem may be security related then this is the defect I opened for some other potentially problematic areas in the security impl that could suffer the same problem that I fixed in 31907. eg. http://issues.apache.org/bugzilla/show_bug.cgi?id=31908 Though the fact that the stack indicates its occurring during the commit would make me think it is defect 32250. Warwick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: major deadlock issues (JDBC store with MySQL)
I recently dropped a defect fix against 2.1b2 for a DB2 client deadlock that was not specific to DB2. The problem was that the server has its own transaction support but some of the operations it was performing on the db weren't being enlisted to the current server trasaction. The defect number for this is 31907 and the patches for it have been submitted to the source tree. You could try pulling down and building the latest source on the 2.1 release branch and running against it. This fix is for clients who are _not_ using client-side transaction support. I have another fix for clients who are using client-side transaction support that I will submit as soon as I am able. Warwick -Original Message- From: Oliver Zeigermann [mailto:[EMAIL PROTECTED] Sent: Saturday, November 20, 2004 2:36 PM To: Slide Users Mailing List Subject: Re: major deadlock issues (JDBC store with MySQL) This should not be caused by using MySQL 4 as the Slide code for both is identical when it comes to inserting content. Additionally, serilzability of requests is handled in the WebDAV layer, so there really should be no deadlocks in the store except there are programming errors (in the store implementation). You do not use externally controlled transactions, do you? Also make sure that there are not open connections with pending transactions from other tools like e.g. an SQL command line tool or a visualizer or anything. Anyway, looking forward to your test case :) Oliver On Sat, 20 Nov 2004 09:26:10 -0800 (PST), [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Thanks a lot for the many updates... I actually don't have access to most of my test data right now, as it is at work. I'll try my JMeter test again on Monday with compression enabled, and I'll also take a look at your Junit test case. The Domain.xml I sent you was indeed the one that was failing. I am using MySQL 4.0.18. We have talked about potentially moving to 4.1 at some point, but for now our production environment depends on 4.0.18. Are there any database settings that I should tweak (e.g. regarding locking?). -Mirko Created a testcase where many threads effectively at random do get,put,delete on the *same* file. I guess this is the hardest concurrency test, the prorgrammer's night mare, right? And it worked with both the 2.1 locking and the new one in 2.2. Only thing I did was to reduce the number of connections to 10 as I do not know how many internal conncections MySQL has by default and how it reacts when it runs out of connections. Some dbs throw errors instead of waiting then... Anyway, the test is at testsuite\testsuite\junit\xmltestcases\functional\extra\multi-user\g etPut\getPutSame.xml in cvs head. Maybe you can tune it to reveal the deadlocks. I will have an additional look at it then. Oliver On Sat, 20 Nov 2004 14:42:44 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: ... and if you provided a test case that revealed any more deadlock I promise that I will try to help you further. Oliver On Sat, 20 Nov 2004 14:40:47 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: And finally took the time to check it with MySQL 4.1 with the attached Domain.xml which differs from yours in details only and: no deadlocks, no conflicts, no problems. Could you describe how your test scenario looks like? Is your attached Domain.xml really the one you used for testing? Oliver On Sat, 20 Nov 2004 13:27:36 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: I just tried this TProcessor.CMD -pattern *cases\\functional\\extra\\multi-user* -users 20 -iterations 10 which is 20 threads repeating uploads, deletes for 10 times and meanwhile browsed Slide with the Windows explorer and it worked without deadlocks. Caching was turned off as well. However, my store is the tx file one and I can not exclude there are still programming errors in the MySQL store as I have never used it. Lately problems were reported with other db stores when not using compression mode. Could you try it again with compression turned on and see if this changes anything. Oliver On Sat, 20 Nov 2004 09:31:17 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I
Re: major deadlock issues (JDBC store with MySQL)
OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /nodestore sequencestore classname=org.apache.slide.store.txfile.FileSequenceStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/sequence/parameter /sequencestore securitystore reference store=nodestore/ /securitystore lockstore reference store=nodestore/ /lockstore revisiondescriptorsstore reference store=nodestore/ /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore/ /revisiondescriptorstore contentstore classname=org.apache.slide.store.txfile.TxFileContentStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/content/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/content/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /contentstore !-- uncomment if you want to use the sample Indexer -- !-- be sure to have Lucene in your classpath -- !-- contentindexer classname=org.apache.slide.index.SampleTxtContainsIndexer parameter name=indexpath./index/parameter /contentindexer -- /store store name=db !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter nodestore classname=org.apache.slide.store.impl.rdbms.JDBCStore parameter name=adapterorg.apache.slide.store.impl.rdbms.MySqlRDBMSAdapter/parameter parameter name=drivercom.mysql.jdbc.Driver/parameter parameter name=urljdbc:mysql://localhost/slide/parameter parameter name=userslide/parameter parameter name=passwordslide/parameter parameter name=dbcpPoolingtrue/parameter parameter name=maxPooledConnections30/parameter parameter name=isolationSERIALIZABLE/parameter parameter name=compressfalse/parameter /nodestore contentstore reference store=nodestore / /contentstore securitystore reference store=nodestore / /securitystore lockstore reference store=nodestore / /lockstore revisiondescriptorsstore reference store=nodestore / /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore / /revisiondescriptorstore /store scope match=/ store=tx/ scope match=/db store=db/ /definition configuration !-- Actions mapping -- read-object/actions/read/read-object create-object/actions/write/create-object remove-object/actions/write/remove-object grant-permission/actions/write-acl/grant-permission revoke-permission/actions/write-acl/revoke-permission read-permissions/actions/read-acl/read-permissions read-own-permissions/actions/read-current-user-privilege-set/read-own-permissions lock-object/actions/write/lock-object kill-lock/actions/unlock/kill-lock read-locks/actions/read/read-locks read-revision-metadata/actions/read/read-revision-metadata create-revision-metadata/actions/write-properties/create-revision-metadata modify-revision-metadata/actions/write-properties/modify-revision-metadata remove-revision-metadata/actions/write-properties/remove-revision-metadata read-revision-content/actions/read/read-revision-content create-revision-content/actions/write-content/create-revision-content modify-revision-content/actions/write-content/modify-revision-content remove-revision-content/actions/write-content/remove-revision-content bind-member/actions/bind/bind-member unbind-member/actions/unbind/unbind-member !-- Paths configuration
Re: major deadlock issues (JDBC store with MySQL)
I just tried this TProcessor.CMD -pattern *cases\\functional\\extra\\multi-user* -users 20 -iterations 10 which is 20 threads repeating uploads, deletes for 10 times and meanwhile browsed Slide with the Windows explorer and it worked without deadlocks. Caching was turned off as well. However, my store is the tx file one and I can not exclude there are still programming errors in the MySQL store as I have never used it. Lately problems were reported with other db stores when not using compression mode. Could you try it again with compression turned on and see if this changes anything. Oliver On Sat, 20 Nov 2004 09:31:17 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /nodestore sequencestore classname=org.apache.slide.store.txfile.FileSequenceStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/sequence/parameter /sequencestore securitystore reference store=nodestore/ /securitystore lockstore reference store=nodestore/ /lockstore revisiondescriptorsstore reference store=nodestore/ /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore/ /revisiondescriptorstore contentstore classname=org.apache.slide.store.txfile.TxFileContentStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/content/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/content/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /contentstore !-- uncomment if you want to use the sample Indexer -- !-- be sure to have Lucene in your classpath -- !-- contentindexer classname=org.apache.slide.index.SampleTxtContainsIndexer parameter name=indexpath./index/parameter /contentindexer -- /store store name=db !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter nodestore classname=org.apache.slide.store.impl.rdbms.JDBCStore parameter name=adapterorg.apache.slide.store.impl.rdbms.MySqlRDBMSAdapter/parameter parameter name=drivercom.mysql.jdbc.Driver/parameter parameter name=urljdbc:mysql://localhost/slide/parameter parameter name=userslide/parameter parameter name=passwordslide/parameter parameter name=dbcpPoolingtrue/parameter parameter name=maxPooledConnections30/parameter parameter name=isolationSERIALIZABLE/parameter parameter name=compressfalse/parameter /nodestore contentstore reference store=nodestore / /contentstore securitystore reference store=nodestore / /securitystore lockstore reference store=nodestore / /lockstore revisiondescriptorsstore reference store=nodestore / /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore / /revisiondescriptorstore /store scope match=/ store=tx/ scope match=/db store=db/ /definition configuration !-- Actions mapping -- read-object/actions/read/read-object create-object/actions/write/create-object remove-object/actions/write/remove-object grant-permission/actions/write-acl/grant-permission revoke-permission/actions/write-acl/revoke-permission read-permissions/actions/read-acl/read-permissions read-own-permissions/actions/read-current-user-privilege-set/read-own-permissions
Re: major deadlock issues (JDBC store with MySQL)
... and if you provided a test case that revealed any more deadlock I promise that I will try to help you further. Oliver On Sat, 20 Nov 2004 14:40:47 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: And finally took the time to check it with MySQL 4.1 with the attached Domain.xml which differs from yours in details only and: no deadlocks, no conflicts, no problems. Could you describe how your test scenario looks like? Is your attached Domain.xml really the one you used for testing? Oliver On Sat, 20 Nov 2004 13:27:36 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: I just tried this TProcessor.CMD -pattern *cases\\functional\\extra\\multi-user* -users 20 -iterations 10 which is 20 threads repeating uploads, deletes for 10 times and meanwhile browsed Slide with the Windows explorer and it worked without deadlocks. Caching was turned off as well. However, my store is the tx file one and I can not exclude there are still programming errors in the MySQL store as I have never used it. Lately problems were reported with other db stores when not using compression mode. Could you try it again with compression turned on and see if this changes anything. Oliver On Sat, 20 Nov 2004 09:31:17 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /nodestore sequencestore classname=org.apache.slide.store.txfile.FileSequenceStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/sequence/parameter /sequencestore securitystore reference store=nodestore/ /securitystore lockstore reference store=nodestore/ /lockstore revisiondescriptorsstore reference store=nodestore/ /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore/ /revisiondescriptorstore contentstore classname=org.apache.slide.store.txfile.TxFileContentStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/content/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/content/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /contentstore !-- uncomment if you want to use the sample Indexer -- !-- be sure to have Lucene in your classpath -- !-- contentindexer classname=org.apache.slide.index.SampleTxtContainsIndexer parameter name=indexpath./index/parameter /contentindexer -- /store store name=db !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter nodestore classname=org.apache.slide.store.impl.rdbms.JDBCStore parameter name=adapterorg.apache.slide.store.impl.rdbms.MySqlRDBMSAdapter/parameter parameter name=drivercom.mysql.jdbc.Driver/parameter parameter name=urljdbc:mysql://localhost/slide/parameter parameter name=userslide/parameter parameter name=passwordslide/parameter parameter name=dbcpPoolingtrue/parameter parameter name=maxPooledConnections30/parameter parameter name=isolationSERIALIZABLE/parameter parameter name=compressfalse/parameter /nodestore contentstore reference
Re: major deadlock issues (JDBC store with MySQL)
Created a testcase where many threads effectively at random do get,put,delete on the *same* file. I guess this is the hardest concurrency test, the prorgrammer's night mare, right? And it worked with both the 2.1 locking and the new one in 2.2. Only thing I did was to reduce the number of connections to 10 as I do not know how many internal conncections MySQL has by default and how it reacts when it runs out of connections. Some dbs throw errors instead of waiting then... Anyway, the test is at testsuite\testsuite\junit\xmltestcases\functional\extra\multi-user\getPut\getPutSame.xml in cvs head. Maybe you can tune it to reveal the deadlocks. I will have an additional look at it then. Oliver On Sat, 20 Nov 2004 14:42:44 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: ... and if you provided a test case that revealed any more deadlock I promise that I will try to help you further. Oliver On Sat, 20 Nov 2004 14:40:47 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: And finally took the time to check it with MySQL 4.1 with the attached Domain.xml which differs from yours in details only and: no deadlocks, no conflicts, no problems. Could you describe how your test scenario looks like? Is your attached Domain.xml really the one you used for testing? Oliver On Sat, 20 Nov 2004 13:27:36 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: I just tried this TProcessor.CMD -pattern *cases\\functional\\extra\\multi-user* -users 20 -iterations 10 which is 20 threads repeating uploads, deletes for 10 times and meanwhile browsed Slide with the Windows explorer and it worked without deadlocks. Caching was turned off as well. However, my store is the tx file one and I can not exclude there are still programming errors in the MySQL store as I have never used it. Lately problems were reported with other db stores when not using compression mode. Could you try it again with compression turned on and see if this changes anything. Oliver On Sat, 20 Nov 2004 09:31:17 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /nodestore sequencestore classname=org.apache.slide.store.txfile.FileSequenceStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/sequence/parameter /sequencestore securitystore reference store=nodestore/ /securitystore lockstore reference store=nodestore/ /lockstore revisiondescriptorsstore reference store=nodestore/ /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore/ /revisiondescriptorstore contentstore classname=org.apache.slide.store.txfile.TxFileContentStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/content/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/content/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /contentstore !-- uncomment if you want to use the sample Indexer -- !-- be sure to have Lucene in your classpath -- !-- contentindexer classname=org.apache.slide.index.SampleTxtContainsIndexer parameter name=indexpath./index/parameter /contentindexer
Re: major deadlock issues (JDBC store with MySQL)
Thanks a lot for the many updates... I actually don't have access to most of my test data right now, as it is at work. I'll try my JMeter test again on Monday with compression enabled, and I'll also take a look at your Junit test case. The Domain.xml I sent you was indeed the one that was failing. I am using MySQL 4.0.18. We have talked about potentially moving to 4.1 at some point, but for now our production environment depends on 4.0.18. Are there any database settings that I should tweak (e.g. regarding locking?). -Mirko Created a testcase where many threads effectively at random do get,put,delete on the *same* file. I guess this is the hardest concurrency test, the prorgrammer's night mare, right? And it worked with both the 2.1 locking and the new one in 2.2. Only thing I did was to reduce the number of connections to 10 as I do not know how many internal conncections MySQL has by default and how it reacts when it runs out of connections. Some dbs throw errors instead of waiting then... Anyway, the test is at testsuite\testsuite\junit\xmltestcases\functional\extra\multi-user\getPut\getPutSame.xml in cvs head. Maybe you can tune it to reveal the deadlocks. I will have an additional look at it then. Oliver On Sat, 20 Nov 2004 14:42:44 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: ... and if you provided a test case that revealed any more deadlock I promise that I will try to help you further. Oliver On Sat, 20 Nov 2004 14:40:47 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: And finally took the time to check it with MySQL 4.1 with the attached Domain.xml which differs from yours in details only and: no deadlocks, no conflicts, no problems. Could you describe how your test scenario looks like? Is your attached Domain.xml really the one you used for testing? Oliver On Sat, 20 Nov 2004 13:27:36 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: I just tried this TProcessor.CMD -pattern *cases\\functional\\extra\\multi-user* -users 20 -iterations 10 which is 20 threads repeating uploads, deletes for 10 times and meanwhile browsed Slide with the Windows explorer and it worked without deadlocks. Caching was turned off as well. However, my store is the tx file one and I can not exclude there are still programming errors in the MySQL store as I have never used it. Lately problems were reported with other db stores when not using compression mode. Could you try it again with compression turned on and see if this changes anything. Oliver On Sat, 20 Nov 2004 09:31:17 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /nodestore sequencestore classname=org.apache.slide.store.txfile.FileSequenceStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/sequence/parameter /sequencestore securitystore reference store=nodestore/ /securitystore lockstore reference store=nodestore/ /lockstore revisiondescriptorsstore reference store=nodestore/ /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore/ /revisiondescriptorstore contentstore classname=org.apache.slide.store.txfile.TxFileContentStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/content/parameter parameter
Re: major deadlock issues (JDBC store with MySQL)
This should not be caused by using MySQL 4 as the Slide code for both is identical when it comes to inserting content. Additionally, serilzability of requests is handled in the WebDAV layer, so there really should be no deadlocks in the store except there are programming errors (in the store implementation). You do not use externally controlled transactions, do you? Also make sure that there are not open connections with pending transactions from other tools like e.g. an SQL command line tool or a visualizer or anything. Anyway, looking forward to your test case :) Oliver On Sat, 20 Nov 2004 09:26:10 -0800 (PST), [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Thanks a lot for the many updates... I actually don't have access to most of my test data right now, as it is at work. I'll try my JMeter test again on Monday with compression enabled, and I'll also take a look at your Junit test case. The Domain.xml I sent you was indeed the one that was failing. I am using MySQL 4.0.18. We have talked about potentially moving to 4.1 at some point, but for now our production environment depends on 4.0.18. Are there any database settings that I should tweak (e.g. regarding locking?). -Mirko Created a testcase where many threads effectively at random do get,put,delete on the *same* file. I guess this is the hardest concurrency test, the prorgrammer's night mare, right? And it worked with both the 2.1 locking and the new one in 2.2. Only thing I did was to reduce the number of connections to 10 as I do not know how many internal conncections MySQL has by default and how it reacts when it runs out of connections. Some dbs throw errors instead of waiting then... Anyway, the test is at testsuite\testsuite\junit\xmltestcases\functional\extra\multi-user\getPut\getPutSame.xml in cvs head. Maybe you can tune it to reveal the deadlocks. I will have an additional look at it then. Oliver On Sat, 20 Nov 2004 14:42:44 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: ... and if you provided a test case that revealed any more deadlock I promise that I will try to help you further. Oliver On Sat, 20 Nov 2004 14:40:47 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: And finally took the time to check it with MySQL 4.1 with the attached Domain.xml which differs from yours in details only and: no deadlocks, no conflicts, no problems. Could you describe how your test scenario looks like? Is your attached Domain.xml really the one you used for testing? Oliver On Sat, 20 Nov 2004 13:27:36 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: I just tried this TProcessor.CMD -pattern *cases\\functional\\extra\\multi-user* -users 20 -iterations 10 which is 20 threads repeating uploads, deletes for 10 times and meanwhile browsed Slide with the Windows explorer and it worked without deadlocks. Caching was turned off as well. However, my store is the tx file one and I can not exclude there are still programming errors in the MySQL store as I have never used it. Lately problems were reported with other db stores when not using compression mode. Could you try it again with compression turned on and see if this changes anything. Oliver On Sat, 20 Nov 2004 09:31:17 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: OK, thanks for providing that info. I will have a look at it ASAP. Which MySQL version are you using? Oliver On Fri, 19 Nov 2004 17:44:11 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter
Re: major deadlock issues (JDBC store with MySQL)
Please send your Domain.xml and the stack traces of said exceptions. You said your were using 2.1b2? Oliver On Fri, 19 Nov 2004 15:59:22 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I did some basic JMeter concurrency testing to validate if Slide meets our concurrency requirements. Unfortunately, the results were not very promising. As I mentioned, we are planning on using Slide as a repository for application data. We expect to have a few hundred concurrent users of our application who would be reading from as well as (although less frequently) writing to the repository. In case it matters, most of the read and write access would be to a dedicated folder per user, rather than a shared folder. I realize that this use case is probably very different from the way most people use Slide, which I assume is mostly for web publishing. My test mainly consists of a JSP page that uses the WebDAV client library to access a folder, iterate over the 10 contained documents and retrieve their contents (about 2k each), and create a new document, retrieve it, and then delete it. I have 100 different folders, all populated with the same 10 documents. My JMeter test consists of 5 concurrent threads that each access a different one of the 100 folders, with 800ms between requests. After only a few seconds, Slide completely deadlocks with various exceptions to that effect in the Tomcat logs. I have tried increasing my JDBC store's MySQL connection pool from 10 to 30, without any improvements. It seems like creating documents is responsible for the deadlocks, as the test behaved fine when it was limited to read-only access. Can anyone think of a way for me to tune the system in order to prevent the deadlocks from occurring? -Mirko - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: major deadlock issues (JDBC store with MySQL)
Make sure you have parameter name=sequential-modefull/parameter parameter name=all-methods-in-transactionstrue/parameter in the configuration sections of Domain.xml which is default for Slide 2.1b2. I have run concurrent tests with 20 threads all writing to the same collection resp. to different ones. One problems Oliver On Sat, 20 Nov 2004 01:05:54 +0100, Oliver Zeigermann [EMAIL PROTECTED] wrote: Please send your Domain.xml and the stack traces of said exceptions. You said your were using 2.1b2? Oliver On Fri, 19 Nov 2004 15:59:22 -0800, Mirko Froehlich [EMAIL PROTECTED] wrote: I did some basic JMeter concurrency testing to validate if Slide meets our concurrency requirements. Unfortunately, the results were not very promising. As I mentioned, we are planning on using Slide as a repository for application data. We expect to have a few hundred concurrent users of our application who would be reading from as well as (although less frequently) writing to the repository. In case it matters, most of the read and write access would be to a dedicated folder per user, rather than a shared folder. I realize that this use case is probably very different from the way most people use Slide, which I assume is mostly for web publishing. My test mainly consists of a JSP page that uses the WebDAV client library to access a folder, iterate over the 10 contained documents and retrieve their contents (about 2k each), and create a new document, retrieve it, and then delete it. I have 100 different folders, all populated with the same 10 documents. My JMeter test consists of 5 concurrent threads that each access a different one of the 100 folders, with 800ms between requests. After only a few seconds, Slide completely deadlocks with various exceptions to that effect in the Tomcat logs. I have tried increasing my JDBC store's MySQL connection pool from 10 to 30, without any improvements. It seems like creating documents is responsible for the deadlocks, as the test behaved fine when it was limited to read-only access. Can anyone think of a way for me to tune the system in order to prevent the deadlocks from occurring? -Mirko - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: major deadlock issues (JDBC store with MySQL)
Yes, I'm using 2.1b2. I have copied my domain.xml below. It does have the two parameters you mentioned. One thing I should mention is that I disabled caching, because I really wanted to simulate the effect of iterating over new content without the effects of caching. My domain.xml defines both a text and a JDBC store, but the JDBC store is the one I'm using for this test. I have also included one of the stack traces below. Domain.xml: = ?xml version=1.0? slide namespace name=slide definition store name=tx !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter parameter name=tlock-timeout120/parameter nodestore classname=org.apache.slide.store.txfile.TxXMLFileDescriptorsStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/metadata/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/metadata/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /nodestore sequencestore classname=org.apache.slide.store.txfile.FileSequenceStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/sequence/parameter /sequencestore securitystore reference store=nodestore/ /securitystore lockstore reference store=nodestore/ /lockstore revisiondescriptorsstore reference store=nodestore/ /revisiondescriptorsstore revisiondescriptorstore reference store=nodestore/ /revisiondescriptorstore contentstore classname=org.apache.slide.store.txfile.TxFileContentStore parameter name=rootpath/usr/local/jakarta-slide/repository/store/content/parameter parameter name=workpath/usr/local/jakarta-slide/repository/work/content/parameter parameter name=defer-savingtrue/parameter parameter name=timeout120/parameter /contentstore !-- uncomment if you want to use the sample Indexer -- !-- be sure to have Lucene in your classpath -- !-- contentindexer classname=org.apache.slide.index.SampleTxtContainsIndexer parameter name=indexpath./index/parameter /contentindexer -- /store store name=db !-- cache settings (added by mirko) -- parameter name=cache-modeoff/parameter parameter name=enable-content-cachingfalse/parameter parameter name=content-cache-size1000/parameter parameter name=object-cache-size1000/parameter nodestore classname=org.apache.slide.store.impl.rdbms.JDBCStore parameter name=adapterorg.apache.slide.store.impl.rdbms.MySqlRDBMSAdapter/parameter parameter name=drivercom.mysql.jdbc.Driver/parameter parameter name=urljdbc:mysql://localhost/slide/parameter parameter name=userslide/parameter parameter name=passwordslide/parameter parameter name=dbcpPoolingtrue/parameter parameter name=maxPooledConnections30/parameter parameter name=isolationSERIALIZABLE/parameter parameter name=compressfalse/parameter /nodestore contentstore reference
