[
https://issues.apache.org/jira/browse/JCR-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499128
]
Ian Boston commented on JCR-929:
--------------------------------
Doing that causes the ClusterNode thread to timeout but the 2 http threads go
into waiting state, so they must be interlocking on waits somewhere.
I cant really do an attempt in the main threads since if the operation doesnt
get to the journal then it wont propagate, and doing a back off would have to
back off far enough to undo the interlock.
The pattern looks slightly different now, to I might have fixed the first
problem.... sorry about all the long stack traces but its going to be
confusing if not in context.
Starting Thread Monitor ==================
Thread Transient File Reaper waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.jackrabbit.util.TransientFileFactory$ReaperThread.run(TransientFileFactory.java:148)
Thread Transient File Reaper waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.jackrabbit.util.TransientFileFactory$ReaperThread.run(TransientFileFactory.java:148)
Thread TP-Processor3 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread TP-Processor2 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread TP-Processor1 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor25 waiting by [EMAIL PROTECTED] ::WAITING at
EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
at java.lang.Object.wait(Object.java:-2)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.nodeAdded(LockManagerImpl.java:840)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.onEvent(LockManagerImpl.java:745)
at
org.apache.jackrabbit.core.observation.EventConsumer.consumeEvents(EventConsumer.java:231)
at
org.apache.jackrabbit.core.observation.ObservationDispatcher.dispatchEvents(ObservationDispatcher.java:201)
at
org.apache.jackrabbit.core.observation.EventStateCollection.dispatch(EventStateCollection.java:424)
at
org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:721)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:855)
at
org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326)
at
org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:313)
at
org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:302)
at
org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:306)
at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1214)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.addMember(DavResourceImpl.java:517)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.doPut(AbstractWebdavServlet.java:504)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:241)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor23 waiting by [EMAIL PROTECTED] ::WAITING at
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(null:-1)
at java.lang.Object.wait(Object.java:-2)
at java.lang.Object.wait(Object.java:474)
at
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(null:-1)
at
org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync(AbstractJournal.java:233)
at
org.apache.jackrabbit.core.journal.DefaultRecordProducer.append(DefaultRecordProducer.java:51)
at
org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceLockChannel.unlocked(ClusterNode.java:637)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.internalUnlock(LockManagerImpl.java:338)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.unlock(LockManagerImpl.java:428)
at
org.apache.jackrabbit.core.lock.XALockManager.unlock(XALockManager.java:103)
at org.apache.jackrabbit.core.NodeImpl.unlock(NodeImpl.java:4133)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.unlock(DavResourceImpl.java:739)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.doUnlock(AbstractWebdavServlet.java:710)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:262)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor22 waiting by [EMAIL PROTECTED] ::WAITING at
EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
at java.lang.Object.wait(Object.java:-2)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.getLockInfo(LockManagerImpl.java:356)
at
org.apache.jackrabbit.core.lock.XALockManager.isLocked(XALockManager.java:143)
at org.apache.jackrabbit.core.NodeImpl.isLocked(NodeImpl.java:4161)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.getLock(DavResourceImpl.java:648)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.initProperties(DavResourceImpl.java:312)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.getProperties(DavResourceImpl.java:271)
at
org.apache.jackrabbit.webdav.MultiStatusResponse.<init>(MultiStatusResponse.java:180)
at
org.apache.jackrabbit.webdav.MultiStatus.addResourceProperties(MultiStatus.java:62)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.doPropFind(AbstractWebdavServlet.java:435)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:232)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor21 waiting by [EMAIL PROTECTED] ::WAITING at
EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
at java.lang.Object.wait(Object.java:-2)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
at
org.apache.jackrabbit.core.lock.LockManagerImpl.getLockInfo(LockManagerImpl.java:356)
at
org.apache.jackrabbit.core.lock.XALockManager.isLocked(XALockManager.java:143)
at org.apache.jackrabbit.core.NodeImpl.isLocked(NodeImpl.java:4161)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.getLock(DavResourceImpl.java:648)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.isLocked(DavResourceImpl.java:855)
at
org.apache.jackrabbit.webdav.simple.DavResourceImpl.addMember(DavResourceImpl.java:501)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.doPut(AbstractWebdavServlet.java:504)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:241)
at
org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor20 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor19 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor18 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor17 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor16 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor15 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor14 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor13 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor12 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor11 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor10 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor9 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor8 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor7 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor6 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor5 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor4 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor3 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor2 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor1 waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread IndexMerger waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread ObservationManager waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread ObservationManager waiting by [EMAIL PROTECTED] ::WAITING at
org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread Finalizer waiting by [EMAIL PROTECTED] ::WAITING
Thread Reference Handler waiting by [EMAIL PROTECTED] ::WAITING
Done Thread Monitor ==================
> Under Heavy load in a Cluster HTTP Threads Block and stall requests
> -------------------------------------------------------------------
>
> Key: JCR-929
> URL: https://issues.apache.org/jira/browse/JCR-929
> Project: Jackrabbit
> Issue Type: Bug
> Components: core
> Affects Versions: 1.3
> Environment: 2 Node Cluster, OSX, JDK 1.5 with DatabaseJournal,
> DatabasePersistanceManager, all content in DB, using WebDAV to load
> Reporter: Ian Boston
> Assigned To: Dominique Pfister
>
> Under Heavy load created by mounting both nodes in the cluster in OSX Finder
> and then uploading large numebers of files to each node at the same time ( a
> few 1000), eventually one of the nodes stops responding and the Finder mount
> timesout and disconnects.
> Once that happens that node becomes unusable.
> More mount attempts will prompt for a password indicating HTTP is still
> running, but will timeout once the connection is authenticated.
> Access by the Web Browser will prompt for a password, conenct and provide a
> once only listing of any collection in the workspace. If you try to refresh
> that collection, the HTTP request hangs forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
