[
https://issues.apache.org/jira/browse/GEODE-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Smith reassigned GEODE-2808:
--------------------------------
Assignee: Dan Smith
> Deadlock in tomcat session module during session expiration
> -----------------------------------------------------------
>
> Key: GEODE-2808
> URL: https://issues.apache.org/jira/browse/GEODE-2808
> Project: Geode
> Issue Type: Bug
> Components: http session
> Reporter: Dan Smith
> Assignee: Dan Smith
>
> We observed a deadlock in the tomcat session state replication module due to
> the way we do expiration. Here are the threads that are involved:
> {noformat}
> Found one Java-level deadlock:
> =============================
> "http-nio-21064-exec-2":
> waiting to lock monitor 0x00007f9a04bde298 (object 0x00000007947da1f0, a
> org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey),
> which is held by "Timer-1"
> "Timer-1":
> waiting to lock monitor 0x00007f99a4004e38 (object 0x00000007947da240, a
> org.apache.geode.modules.session.catalina.DeltaSession8),
> which is held by "http-nio-21064-exec-2"
> Java stack information for the threads listed above:
> ===================================================
> "http-nio-21064-exec-2":
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1228)
> - waiting to lock <0x00000007947da1f0> (a
> org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6785)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6762)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6724)
> at
> org.apache.geode.internal.cache.LocalRegion.validatedDestroy(LocalRegion.java:1110)
> at
> org.apache.geode.internal.cache.LocalRegion.destroy(LocalRegion.java:1095)
> at
> org.apache.geode.internal.cache.AbstractRegion.destroy(AbstractRegion.java:271)
> at
> org.apache.geode.modules.session.catalina.AbstractSessionCache.destroySession(AbstractSessionCache.java:72)
> at
> org.apache.geode.modules.session.catalina.DeltaSessionManager.remove(DeltaSessionManager.java:405)
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:850)
> - locked <0x00000007947da240> (a
> org.apache.geode.modules.session.catalina.DeltaSession8)
> at
> org.apache.catalina.session.StandardSession.isValid(StandardSession.java:682)
> at org.apache.catalina.connector.Request.doGetSession(Request.java:2917)
> at org.apache.catalina.connector.Request.getSession(Request.java:2367)
> at
> org.apache.geode.modules.session.catalina.CommitSessionValve.invoke(CommitSessionValve.java:50)
> at
> org.apache.geode.modules.session.catalina.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:46)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
> at
> org.apache.geode.modules.session.catalina.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:46)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
> at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> - locked <0x00000007948d6870> (a
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> "Timer-1":
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:791)
> - waiting to lock <0x00000007947da240> (a
> org.apache.geode.modules.session.catalina.DeltaSession8)
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:766)
> at
> org.apache.geode.modules.session.catalina.DeltaSession8.processExpired(DeltaSession8.java:320)
> at
> org.apache.geode.modules.session.catalina.callback.SessionExpirationCacheListener.afterDestroy(SessionExpirationCacheListener.java:56)
> at
> org.apache.geode.internal.cache.EnumListenerEvent$AFTER_DESTROY.dispatchEvent(EnumListenerEvent.java:151)
> at
> org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8751)
> at
> org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:7245)
> at
> org.apache.geode.internal.cache.LocalRegion.invokeDestroyCallbacks(LocalRegion.java:7052)
> at
> org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2293)
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:140)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroyPart2(LocalRegion.java:6992)
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1419)
> - locked <0x00000007947da1f0> (a
> org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6785)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6762)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6724)
> at
> org.apache.geode.internal.cache.LocalRegion.expireDestroy(LocalRegion.java:6864)
> at
> org.apache.geode.internal.cache.EntryExpiryTask.destroy(EntryExpiryTask.java:119)
> at
> org.apache.geode.internal.cache.ExpiryTask.expire(ExpiryTask.java:278)
> at
> org.apache.geode.internal.cache.ExpiryTask.expire(ExpiryTask.java:232)
> at
> org.apache.geode.internal.cache.EntryExpiryTask.basicPerformTimeout(EntryExpiryTask.java:305)
> at
> org.apache.geode.internal.cache.LocalRegion.performExpiryTimeout(LocalRegion.java:801)
> at
> org.apache.geode.internal.cache.EntryExpiryTask.performTimeout(EntryExpiryTask.java:219)
> at
> org.apache.geode.internal.cache.ExpiryTask.runInThreadPool(ExpiryTask.java:370)
> at org.apache.geode.internal.cache.ExpiryTask.run2(ExpiryTask.java:318)
> at
> org.apache.geode.internal.SystemTimer$SystemTimerTask.run(SystemTimer.java:444)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Found 1 deadlock.
> {noformat}
> The basic issue is that we configure expiration on the geode region to expire
> sessions, and the tomcat container will also expire sessions when the get to
> old. But those two threads get locks in a different order:
> Tomcat expiration (http-nio-21064-exec-2)
> # sync session in the tomcat code (StandardSession.expire)
> # sync region entry in the geode code (AbstractRegionMap.destroy)
> region expiration
> # sync region entry in the geode code (AbstractRegionMap.destroy
> # sync session in geode code (StandardSession.expire)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
