Hi Devs,
Currently we have significant amount of read write locks in Stratos
codebase to synchronize data structure updates made by multiple threads.
As I figured out recently there are several possibilities that these locks
could cause problems if not properly implemented:
*Problem 1: A thread trying to acquire a write lock while having a read
lock.*
*Problem 2: Locks acquired but not released*
I just implemented a ReadWriteLock class in stratos common module which
could detect these problems. This class make use
of java.util.concurrent.locks.ReentrantReadWriteLock and implements
additional logic to detect problems. By default problem 2 detection logic
is disabled, it can be enabled by setting following system property:
*read.write.lock.monitor.enabled=true*
I have now updated all read write locks with the above class.
Following is an example scenario where I detected an unreleased lock in the
REST API:
org.apache.stratos.common.exception.LockNotReleasedException
at
org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
at
org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-01-20 18:30:12,952] ERROR
{org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
error, lock has not released for 30 seconds: [lock-name] topology
[lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
[stack-trace]
java.lang.Thread.getStackTrace(Thread.java:1589)
org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
Thanks
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
