On Fri, Nov 15, 2013 at 12:29 PM, Miyuru Wanninayaka <miy...@wso2.com>wrote:
> Seems once proxy service deploy/undploy lock in axis2 config instead of
> synapse config, no more deadlocks :)
>
> I have to
>
> - private AxisConfiguration axisConfiguration = null;
> + private AxisConfiguration axisConfiguration = new AxisConfiguration();
>
> because in MultiXMLConfigurationBuilderTest, it creates a synapse config
> without axisConfig which will cause NPE in synchronization block.
>
> I'm currently running tests with these changes.
>
>
> On Fri, Nov 15, 2013 at 10:37 AM, Miyuru Wanninayaka <miy...@wso2.com>wrote:
>
>> May be for proxy service undeployment, synapse can lock on axis2 config
>> instead of synapse config, which will avoid deadlocking.
>>
>>
>> On Fri, Nov 15, 2013 at 10:22 AM, Miyuru Wanninayaka <miy...@wso2.com>wrote:
>>
>>> I have observed this few times even without patch0002.
>>>
>>> When analyzed deadloak flow it seems
>>> hot deployment of main, fault sequences and deleting ESB artifacts from
>>> admin services triggers in parallel.
>>>
>>> in hot deployment, axis2 thread first get a lock of axis2 configuration
>>> and then calls synapse deployer.undeploy which will lock on synapse
>>> configuration.
>>> during admin service call, it will directly call synapse to undeploy
>>> artifacts which will first lock on synapse config and in case of proxy, it
>>> will cause related axis2 service to undeploy and try to acquire lock of
>>> axis2 config which will become a deadlock.
>>>
>>> I have modified abstract ESBIntegrationTest.cleanup() method to wait few
>>> seconds to let main and fault sequence to hot update. Seems deadlock does
>>> not happens after that.
>>>
>>>
>>> On Thu, Nov 14, 2013 at 10:17 PM, Miyuru Wanninayaka <miy...@wso2.com>wrote:
>>>
>>>> Seems patch0002 causing deadlock. Once patch0002 removed from pack,
>>>> tests are running without any issue.
>>>>
>>>>
>>>> On Thu, Nov 14, 2013 at 9:31 PM, Krishantha Samaraweera <
>>>> krishan...@wso2.com> wrote:
>>>>
>>>>> Integration tests hang at the early stage and following dead lock
>>>>> situation detected in ESB Alpha pack.
>>>>>
>>>>> Name: http-nio-9443-exec-19
>>>>> State: BLOCKED on org.apache.axis2.engine.AxisConfiguration@510b02daowned
>>>>> by: pool-15-thread-1
>>>>> Total blocked: 2 Total waited: 29
>>>>>
>>>>> Stack trace:
>>>>>
>>>>>
>>>>> org.apache.axis2.engine.AxisConfiguration.removeService(AxisConfiguration.java:692)
>>>>>
>>>>> org.apache.synapse.config.SynapseConfiguration.removeProxyService(SynapseConfiguration.java:1037)
>>>>> - locked org.apache.synapse.config.SynapseConfiguration@73e64d34
>>>>>
>>>>> org.wso2.carbon.proxyadmin.service.ProxyServiceAdmin.deleteProxyService(ProxyServiceAdmin.java:398)
>>>>> sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> java.lang.reflect.Method.invoke(Method.java:597)
>>>>>
>>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>>
>>>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
>>>>>
>>>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>>>>>
>>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>>>
>>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>>>
>>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>>>
>>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:
>>>>>
>>>>>
>>>>> ---------------------
>>>>>
>>>>> Name: pool-15-thread-1
>>>>> State: BLOCKED on
>>>>> org.apache.synapse.config.SynapseConfiguration@73e64d34 owned by:
>>>>> http-nio-9443-exec-19
>>>>> Total blocked: 1 Total waited: 17
>>>>>
>>>>> Stack trace:
>>>>>
>>>>>
>>>>> org.apache.synapse.config.SynapseConfiguration.getDefinedSequences(SynapseConfiguration.java:289)
>>>>>
>>>>> org.apache.synapse.deployers.SequenceDeployer.updateSynapseArtifact(SequenceDeployer.java:120)
>>>>>
>>>>> org.wso2.carbon.sequences.SequenceDeploymentInterceptor.updateSynapseArtifact(SequenceDeploymentInterceptor.java:49)
>>>>>
>>>>> org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.resetDefaultSequence(AbstractSynapseArtifactDeployer.java:584)
>>>>>
>>>>> org.apache.synapse.deployers.SequenceDeployer.updatingFaultAndMainWithOriginal(SequenceDeployer.java:181)
>>>>>
>>>>> org.apache.synapse.deployers.SequenceDeployer.undeploy(SequenceDeployer.java:150)
>>>>>
>>>>> org.apache.axis2.deployment.DeploymentEngine.unDeploy(DeploymentEngine.java:909)
>>>>> - locked org.wso2.carbon.core.CarbonAxisConfigurator@5497eb53
>>>>>
>>>>> org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:143)
>>>>> - locked org.wso2.carbon.core.CarbonAxisConfigurator@5497eb53
>>>>>
>>>>> org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
>>>>>
>>>>> org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
>>>>>
>>>>> org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
>>>>>
>>>>> --
>>>>> Krishantha Samaraweera
>>>>> Senior Technical Lead - Test Automation
>>>>> Mobile: +94 77 7759918
>>>>> WSO2, Inc.; http://wso2.com/
>>>>> lean . enterprise . middlewear.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Miyuru Wanninayaka
>>>> Technical Lead
>>>> WSO2 Inc. : http://wso2.com
>>>>
>>>> Mobile : +94 77 209 9788
>>>> Blog : http://miyurudw.blogspot.com
>>>> Flickr : http://www.flickr.com/photos/miyuru_daminda
>>>>
>>>
>>>
>>>
>>> --
>>> Miyuru Wanninayaka
>>> Technical Lead
>>> WSO2 Inc. : http://wso2.com
>>>
>>> Mobile : +94 77 209 9788
>>> Blog : http://miyurudw.blogspot.com
>>> Flickr : http://www.flickr.com/photos/miyuru_daminda
>>>
>>
>>
>>
>> --
>> Miyuru Wanninayaka
>> Technical Lead
>> WSO2 Inc. : http://wso2.com
>>
>> Mobile : +94 77 209 9788
>> Blog : http://miyurudw.blogspot.com
>> Flickr : http://www.flickr.com/photos/miyuru_daminda
>>
>
>
>
> --
> Miyuru Wanninayaka
> Technical Lead
> WSO2 Inc. : http://wso2.com
>
> Mobile : +94 77 209 9788
> Blog : http://miyurudw.blogspot.com
> Flickr : http://www.flickr.com/photos/miyuru_daminda
>
--
Miyuru Wanninayaka
Technical Lead
WSO2 Inc. : http://wso2.com
Mobile : +94 77 209 9788
Blog : http://miyurudw.blogspot.com
Flickr : http://www.flickr.com/photos/miyuru_daminda
Index: src/main/java/org/apache/synapse/config/SynapseConfiguration.java
===================================================================
--- src/main/java/org/apache/synapse/config/SynapseConfiguration.java
(revision 191088)
+++ src/main/java/org/apache/synapse/config/SynapseConfiguration.java
(working copy)
@@ -136,7 +136,7 @@
/**
* Hold reference to the Axis2 ConfigurationContext
*/
- private AxisConfiguration axisConfiguration = null;
+ private AxisConfiguration axisConfiguration = new AxisConfiguration();
/**
* Save the path to the configuration file loaded, to save it later if
@@ -995,14 +995,17 @@
* @param name the name of the Proxy service
* @param proxy the Proxy service instance
*/
- public synchronized void addProxyService(String name, ProxyService proxy) {
- if (!proxyServices.containsKey(name)) {
- proxyServices.put(name, proxy);
- for (SynapseObserver o : observers) {
- o.proxyServiceAdded(proxy);
+ public void addProxyService(String name, ProxyService proxy) {
+ synchronized (this.axisConfiguration) {
+ log.info(axisConfiguration == null);
+ if (!proxyServices.containsKey(name)) {
+ proxyServices.put(name, proxy);
+ for (SynapseObserver o : observers) {
+ o.proxyServiceAdded(proxy);
+ }
+ } else {
+ handleException("Duplicate proxy service by the name : " +
name);
}
- } else {
- handleException("Duplicate proxy service by the name : " + name);
}
}
@@ -1022,26 +1025,28 @@
*
* @param name of the Proxy Service to be deleted
*/
- public synchronized void removeProxyService(String name) {
- ProxyService proxy = proxyServices.get(name);
- if (proxy == null) {
- handleException("Unknown proxy service for name : " + name);
- } else {
- try {
- if (getAxisConfiguration().getServiceForActivation(name) !=
null) {
- if (getAxisConfiguration().getServiceForActivation(name)
- .isActive()) {
- getAxisConfiguration().getService(name)
- .setActive(false);
+ public void removeProxyService(String name) {
+ synchronized (this.axisConfiguration) {
+ ProxyService proxy = proxyServices.get(name);
+ if (proxy == null) {
+ handleException("Unknown proxy service for name : " + name);
+ } else {
+ try {
+ if (getAxisConfiguration().getServiceForActivation(name)
!= null) {
+ if
(getAxisConfiguration().getServiceForActivation(name)
+ .isActive()) {
+ getAxisConfiguration().getService(name)
+ .setActive(false);
+ }
+ getAxisConfiguration().removeService(name);
}
- getAxisConfiguration().removeService(name);
+ proxyServices.remove(name);
+ for (SynapseObserver o : observers) {
+ o.proxyServiceRemoved(proxy);
+ }
+ } catch (AxisFault axisFault) {
+ handleException(axisFault.getMessage());
}
- proxyServices.remove(name);
- for (SynapseObserver o : observers) {
- o.proxyServiceRemoved(proxy);
- }
- } catch (AxisFault axisFault) {
- handleException(axisFault.getMessage());
}
}
}
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
