Hi Imesh, As per the discussion in the thread, the final fix I did was removing the logic that is doing the validation is parallel and doing it in sequentially.
On Tue, Nov 10, 2015 at 3:52 AM, Imesh Gunaratne <im...@apache.org> wrote: > A great finding Isuru! I didn't get the fix you have proposed. Can you > please explain it bit further? > > Thanks > > On Mon, Nov 9, 2015 at 11:32 AM, Isuru Haththotuwa <isu...@apache.org> > wrote: > >> >> >> On Mon, Nov 9, 2015 at 12:17 PM, Akila Ravihansa Perera < >> raviha...@wso2.com> wrote: >> >>> Nice catch Isuru! >>> >>> There is a JIRA already reported [1] for this issue. I've marked it as a >>> duplicate. >>> >>> Shall we get rid of this PartitionValidatorCallable class? I think it >>> adds unnecessary complexity to validation logic. >>> >> Agreed. Will remove this. >> >>> >>> [1] https://issues.apache.org/jira/browse/STRATOS-1595 >>> >>> Thanks. >>> >>> On Mon, Nov 9, 2015 at 11:56 AM, Isuru Haththotuwa <isu...@apache.org> >>> wrote: >>> >>>> Hi Devs, >>>> >>>> The error [1] is reproducible with 4.1.4 Stratos release. This can be >>>> reproduced by undeploying the application + cartridge, and then re-adding >>>> them, followed by deploying them from UI. This is occurring due to a race >>>> condition, when we are validating the partitions in parallel using >>>> PartitionValidatorCallable class. The getIaaS method of IaaSProvider [2], >>>> used in validatePartitionAndGetIaasProvider method of >>>> ClousControllerServiceUtils class [3] is not thread safe. We can use the >>>> buildIaaS method of same class, which is thread safe. I have tested this >>>> fix and it works as expected. Created issue [4] to track. >>>> >>>> [1] TID: [0] [PPAAS] [2015-11-04 18:06:11,169] ERROR >>>> {org.apache.stratos.cloud.controller.iaases.ec2.EC2PartitionValidator} >>>> >>>> - Invalid partition detected: [partition-id] partition-2 >>>> java.lang.NullPointerException >>>> at >>>> >>>> org.apache.stratos.cloud.controller.iaases.ec2.EC2Iaas.isValidRegion(EC2Iaas.java:338) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.iaases.ec2.EC2PartitionValidator.validate(EC2PartitionValidator.java:56) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceUtil.validatePartitionAndGetIaasProvider(CloudControllerServiceUtil.java:82) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.concurrent.PartitionValidatorCallable.call(PartitionValidatorCallable.java:51) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.concurrent.PartitionValidatorCallable.call(PartitionValidatorCallable.java:31) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> TID: [0] [PPAAS] [2015-11-04 18:06:11,170] ERROR >>>> >>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>>> >>>> >>>> - Could not cache partitions against the cartridge: [cartridge-type] >>>> wso2dss-322-manager >>>> java.util.concurrent.ExecutionException: >>>> org.apache.stratos.cloud.controller.exception.InvalidPartitionException: >>>> Invalid partition detected: [partition-id] partition-2 >>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:188) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.validateDeploymentPolicyNetworkPartition(CloudControllerServiceImpl.java:991) >>>> at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source) >>>> at >>>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> >>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>>> at >>>> >>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117) >>>> at >>>> >>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40) >>>> at >>>> >>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>> at >>>> >>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>>> at >>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>>> at >>>> >>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> at >>>> >>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>>> at >>>> >>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) >>>> at >>>> >>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) >>>> at >>>> >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) >>>> at >>>> >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) >>>> at >>>> >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) >>>> at >>>> >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) >>>> at >>>> >>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) >>>> at >>>> >>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) >>>> at >>>> >>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) >>>> at >>>> >>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>>> at >>>> >>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>>> at >>>> >>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>>> at >>>> >>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) >>>> at >>>> >>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>>> at >>>> >>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) >>>> at >>>> >>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) >>>> at >>>> >>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) >>>> at >>>> >>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) >>>> at >>>> >>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1736) >>>> at >>>> >>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1695) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at >>>> >>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: >>>> org.apache.stratos.cloud.controller.exception.InvalidPartitionException: >>>> Invalid partition detected: [partition-id] partition-2 >>>> at >>>> >>>> org.apache.stratos.cloud.controller.iaases.ec2.EC2PartitionValidator.validate(EC2PartitionValidator.java:73) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceUtil.validatePartitionAndGetIaasProvider(CloudControllerServiceUtil.java:82) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.concurrent.PartitionValidatorCallable.call(PartitionValidatorCallable.java:51) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.concurrent.PartitionValidatorCallable.call(PartitionValidatorCallable.java:31) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> ... 1 more >>>> Caused by: java.lang.NullPointerException >>>> at >>>> >>>> org.apache.stratos.cloud.controller.iaases.ec2.EC2Iaas.isValidRegion(EC2Iaas.java:338) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.iaases.ec2.EC2PartitionValidator.validate(EC2PartitionValidator.java:56) >>>> ... 7 more >>>> TID: [0] [PPAAS] [2015-11-04 18:06:11,189] ERROR >>>> {org.apache.stratos.autoscaler.monitor.MonitorFactory} >>>> >>>> >>>> - Error while validating deployment policy from cloud controller >>>> [network-partition-id] network-partition-ec2 >>>> >>>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceInvalidPartitionExceptionException: >>>> CloudControllerServiceInvalidPartitionExceptionException >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>> Method) >>>> at >>>> >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>> at >>>> >>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>> at java.lang.Class.newInstance(Class.java:379) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3091) >>>> at >>>> >>>> org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:193) >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270) >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83) >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> TID: [0] [PPAAS] [2015-11-04 18:06:11,189] ERROR >>>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} >>>> >>>> >>>> - An error occurred while starting monitor: [type] cluster >>>> [component] >>>> wso2dss-322-application.wso2dss-322-manager.wso2dss-322-manager.domain >>>> java.lang.RuntimeException: Error while validating deployment >>>> policy from cloud controller [network-partition-id] >>>> network-partition-ec2 >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:278) >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83) >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: >>>> >>>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceInvalidPartitionExceptionException: >>>> CloudControllerServiceInvalidPartitionExceptionException >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>> Method) >>>> at >>>> >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>> at >>>> >>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>> at java.lang.Class.newInstance(Class.java:379) >>>> at >>>> >>>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3091) >>>> at >>>> >>>> org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:193) >>>> at >>>> >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270) >>>> ... 7 more >>>> >>>> [2]. org.apache.stratos.cloud.controller.domain.IaaSProvider >>>> >>>> [3]. >>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceUtil >>>> >>>> [4]. https://issues.apache.org/jira/browse/STRATOS-1603 >>>> >>>> -- >>>> Thanks and Regards, >>>> >>>> Isuru H. >>>> +94 716 358 048* <http://wso2.com/>* >>>> >>>> >>>> * <http://wso2.com/>* >>>> >>>> >>>> >>> >>> >>> -- >>> Akila Ravihansa Perera >>> WSO2 Inc.; http://wso2.com/ >>> >>> Blog: http://ravihansa3000.blogspot.com >>> >>> -- >>> <http://ravihansa3000.blogspot.com> >>> <http://ravihansa3000.blogspot.com> >>> Thanks and Regards, >>> >>> Isuru H. >>> <http://ravihansa3000.blogspot.com> >>> +94 716 358 048 <http://ravihansa3000.blogspot.com>* <http://wso2.com/>* >>> >>> >>> * <http://wso2.com/>* >>> >>> >>> > > > -- > Imesh Gunaratne > > Senior Technical Lead, WSO2 > Committer & PMC Member, Apache Stratos > > -- > Thanks and Regards, > > Isuru H. > +94 716 358 048* <http://wso2.com/>* > > > * <http://wso2.com/>* > > >
