[
https://issues.apache.org/jira/browse/OAK-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357080#comment-14357080
]
Alex Parvulescu commented on OAK-2560:
--------------------------------------
So this boils down to
{code}
SegmentNodeStoreService#deactivate
- FileStore#close locks [FileStore] with <6800>
- FileStore#flush locks [FileStore#persistedHead] with <6846>
- tracker.getWriter().flush() waits on [SegmentWriter] <680c>
SegmentWriter#writeMap
- SegmentWriter#writeMapBucket locks [SegmentWriter] with <680c>
- SegmentId#getSegment locks [SegmentId] with <6b0b>
- SegmentTracker#getSegment
- FileStore#readSegment waits on [FileStore] <6800>
{code}
but the problem is I'm not really convinced that the lock order is the problem.
I think the problem is the fact that the FileStore is closing and there's no
way to prevent others from continuing with loading segments. So basically a
#readSegment call after a #close call should be considered illegal.
[~mduerig] thoughts?
> SegmentStore deadlock on shutdown
> ---------------------------------
>
> Key: OAK-2560
> URL: https://issues.apache.org/jira/browse/OAK-2560
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Fix For: 1.2
>
>
> We've ran into a deadlock in the Oak SegmentNodeStore between
> _org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.deactivate(SegmentNodeStoreService.java:300)_
> and _
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:120)_
> This was seen on Oak 1.0.11.
> {code}
> Java stack information for the threads listed above
>
> "pool-9-thread-5":
> at
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:656)
> - waiting to lock <0x0000000680024468> (a
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:120)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
> - locked <0x00000006b0b7ae28> (a
> org.apache.jackrabbit.oak.plugins.segment.SegmentId)
> at
> org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:128)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:462)
> - locked <0x0000000680c634a8> (a
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:460)
> - locked <0x0000000680c634a8> (a
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:660)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1020)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1022)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1003)
> at
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:994)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1003)
> at
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:994)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:91)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:77)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:204)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:73)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:331)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:323)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.child(MemoryNodeBuilder.java:311)
> at
> org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.insert(ContentMirrorStoreStrategy.java:112)
> at
> org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.update(ContentMirrorStoreStrategy.java:81)
> at
> org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexEditor.leave(PropertyIndexEditor.java:261)
> at
> org.apache.jackrabbit.oak.spi.commit.CompositeEditor.leave(CompositeEditor.java:74)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:130)
> at
> org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
> at
> org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
> at
> org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
> at
> org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeAdded(MapRecord.java:383)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeAdded(MapRecord.java:425)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:483)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:430)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:430)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:354)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:300)
> - locked <0x000000068afa00e8> (a
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
> at
> org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:105)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> "FelixStartLevel":
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:167)
> - waiting to lock <0x0000000680c634a8> (a
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
> at
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:431)
> - locked <0x000000068461ecb8> (a
> java.util.concurrent.atomic.AtomicReference)
> at
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.close(FileStore.java:582)
> - locked <0x0000000680024468> (a
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.deactivate(SegmentNodeStoreService.java:300)
> - locked <0x00000006800243c8> (a
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)
> at
> org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)
> at
> org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.disposeImplementationObject(SingleComponentManager.java:355)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.deleteComponent(SingleComponentManager.java:170)
> at
> org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:908)
> at
> org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:883)
> at
> org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:580)
> at
> org.apache.felix.scr.impl.config.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:406)
> at
> org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:335)
> at
> org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:313)
> at org.apache.felix.scr.impl.Activator.access$300(Activator.java:45)
> at
> org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:198)
> at
> org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
> at
> org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
> at
> org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:869)
> at
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:790)
> at
> org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:515)
> at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4409)
> at org.apache.felix.framework.Felix.stopBundle(Felix.java:2526)
> at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1315)
> at
> org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
