Hi Raj, Few minutes back in another email I raised an issue of multiple instances of the already Active cartridges getting spawned & shared the wsocarbon.log Actually all of my cartridges are having a static ip allocated to them. So whenever stratos tried to launch/spin another instance of the already active cartridge, it fails with IP/MAC-Address conflict.
[cid:image001.png@01CFFD04.09778CC0] Regards Manas From: Rajkumar Rajaratnam [mailto:rajkum...@wso2.com] Sent: Monday, November 10, 2014 4:13 PM To: Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) Cc: dev@stratos.apache.org Subject: Re: Getting Cluster Monitor Failure Messages in wso2carbon.log Hi Manas, I quickly went through the logs and mincheck.drl. And these are my observations. * Nothing wrong with mincheck.drl * Cluster monitor is created successfully * Cluster monitor is executing monitor() method * monitor() executes mincheck drool, since min rule is not satisfied, it delegates request to RuleTaskDelegator to spawn an instance. * RuleTaskDelegator calls CC to create an instance which is failing[1] * So mincheck rule is failing, implies that monitor is failing So the root cause is instance creation is failing. 1 = > TID: [0] [STRATOS] [2014-11-10 07:24:01,648] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Trying to spawn an instance via cloud controller: [cluster] imc-dm.imc-dm.domain [partition] RegionOne-Core [lb-cluster] null [network-partition-id] N1 {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} TID: [0] [STRATOS] [2014-11-10 07:24:01,651] ERROR {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor: Monitor failed.ClusterMonitor [clusterId=imc-ce.imc-ce.domain, serviceId=imc-ce, deploymentPolicy=Deployment Policy [id]static-1-Core [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@5126e236], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null, hasPrimary=false ] {org.apache.stratos.autoscaler.monitor.ClusterMonitor} Exception executing consequence for rule "Minimum Rule" in org.apache.stratos.autoscaler.rule: [Error: $delegator.delegateSpawn($ctxt, clusterId, lbRef, isPrimary): Cannot spawn an instance] [Near : {... if (isPrimary){ ....}] ^ [Line: 1, Column: 1] at org.drools.runtime.rule.impl.DefaultConsequenceExceptionHandler.handleException(DefaultConsequenceExceptionHandler.java:39) at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1297) at org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221) at org.drools.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1456) at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:710) at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:674) at org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:230) at org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator.evaluateMinCheck(AutoscalerRuleEvaluator.java:94) at org.apache.stratos.autoscaler.monitor.ClusterMonitor.monitor(ClusterMonitor.java:157) at org.apache.stratos.autoscaler.monitor.ClusterMonitor.run(ClusterMonitor.java:86) at java.lang.Thread.run(Thread.java:745) Caused by: [Error: $delegator.delegateSpawn($ctxt, clusterId, lbRef, isPrimary): Cannot spawn an instance] [Near : {... if (isPrimary){ ....}] ^ [Line: 1, Column: 1] at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:416) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.optimizeAccessor(ReflectiveAccessorOptimizer.java:143) at org.mvel2.ast.ASTNode.optimize(ASTNode.java:159) at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:115) at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85) at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113) at org.mvel2.MVEL.executeExpression(MVEL.java:930) at org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104) at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287) ... 9 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1104) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:987) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:377) ... 19 more Caused by: java.lang.RuntimeException: Cannot spawn an instance at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegateSpawn(RuleTasksDelegator.java:107) ... 26 more Caused by: org.apache.stratos.autoscaler.exception.SpawningException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:174) at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegateSpawn(RuleTasksDelegator.java:87) ... 26 more Caused by: org.apache.axis2.AxisFault: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.fromOM(CloudControllerServiceStub.java:4312) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.startInstance(CloudControllerServiceStub.java:1415) at org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:162) ... 27 more Caused by: org.apache.axiom.om.impl.exception.OMStreamingException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at org.apache.axiom.om.impl.SwitchingWrapper.getElementText(SwitchingWrapper.java:966) at javax.xml.stream.util.StreamReaderDelegate.getElementText(StreamReaderDelegate.java:100) at org.apache.stratos.cloud.controller.stub.pojo.MemberContext$Factory.parse(MemberContext.java:1470) at org.apache.stratos.cloud.controller.stub.impl.StartInstanceResponse$Factory.parse(StartInstanceResponse.java:417) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.fromOM(CloudControllerServiceStub.java:4159) ... 29 more Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.getElementText(XMLStreamReaderImpl.java:856) at org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getElementText(XMLStreamReaderWrapper.java:100) at org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getElementText(XMLStreamReaderWrapper.java:100) at org.apache.axiom.om.impl.SwitchingWrapper.getElementText(SwitchingWrapper.java:962) We need to look at why instance creation is failing. Do you have any idea why instance creation is failing? I am looking into it further. Thanks. On Mon, Nov 10, 2014 at 3:31 PM, Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com<mailto:mpade...@cisco.com>> wrote: Hi Raj, Mincheck.drl is attached herewith. Regards Manas From: Rajkumar Rajaratnam [mailto:rajkum...@wso2.com<mailto:rajkum...@wso2.com>] Sent: Monday, November 10, 2014 3:18 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: Re: Getting Cluster Monitor Failure Messages in wso2carbon.log Great Manas. I will go through the log and get back to you. It would be great if you can add the mincheck.drl file too. Thanks. On Mon, Nov 10, 2014 at 3:10 PM, Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com<mailto:mpade...@cisco.com>> wrote: Hi Raj, Attached is the wso2carbon.log as requested by you. Regards Manas From: Rajkumar Rajaratnam [mailto:rajkum...@wso2.com<mailto:rajkum...@wso2.com>] Sent: Monday, October 27, 2014 5:52 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: Re: Getting Cluster Monitor Failure Messages in wso2carbon.log Will you be able to share the complete wso2carbon.log file? (at least from the autoscale policy deployment upto this error message) Thanks. On Mon, Oct 27, 2014 at 5:41 PM, Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com<mailto:mpade...@cisco.com>> wrote: Hi Raj, I am using the autoscale policy name correctly during subscription & cartridge gets spawned, fetches catalog from puppet-master & becoming Active. After this I am getting Cluster monitor: Monitor Failed. If I will use the wrong autoscale-policy name for cartridge subscription, it will fail there only. Regards Manas From: Rajkumar Rajaratnam [mailto:rajkum...@wso2.com<mailto:rajkum...@wso2.com>] Sent: Monday, October 27, 2014 5:20 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: Re: Getting Cluster Monitor Failure Messages in wso2carbon.log oops. so we don't need subscription json to subscribe. I confused with kubernetes stuff. AFAIK, this error happens when the relevant autoscale policy is not found in Policy Manager. When we pass autoscale policy name, AS is loading the autoscale policy from Policy Manager. If the policy with the name you provided is not there, it will give this error. Please make sure that you have given the correct autoscale policy name. Thanks. On Mon, Oct 27, 2014 at 5:00 PM, Rajkumar Rajaratnam <rajkum...@wso2.com<mailto:rajkum...@wso2.com>> wrote: Hi Manas, If possible, can you share your subscription json and autoscale policy json? Thanks. On Mon, Oct 27, 2014 at 4:56 PM, Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com<mailto:mpade...@cisco.com>> wrote: Hi Raj, I am not using Stratos-GUI & working from CLI only. Regards Manas From: Rajkumar Rajaratnam [mailto:rajkum...@wso2.com<mailto:rajkum...@wso2.com>] Sent: Monday, October 27, 2014 4:53 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: Re: Getting Cluster Monitor Failure Messages in wso2carbon.log I assumed you were using CLI or Curl commands. If you are using management console, did you select the autoscale policy from autoscale policies drop down list? And could you please tell me whether you are using master branch or some other branch? Thanks. On Mon, Oct 27, 2014 at 4:42 PM, Rajkumar Rajaratnam <rajkum...@wso2.com<mailto:rajkum...@wso2.com>> wrote: Hi Manas, Reason is your autoscale policy is null. You need to give a deployed autoscale policy name along with subscription json. Thanks. On Mon, Oct 27, 2014 at 4:36 PM, Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com<mailto:mpade...@cisco.com>> wrote: Hi, I am getting following messages periodically in wso2carbon.log for all of the subscribed cartridges which came Active. Want to know what is triggering this error to come & what is its impact if any. TID: [0] [STRATOS] [2014-10-13 13:04:59,436] ERROR {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor: Monitor failed.ClusterMonitor [clusterId=1-14.1-14.domain, serviceId=1-14, deploymentPolicy=Deployment Policy [id]static-1-Core [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@118c06f9], autoscalePolicy=null, lbReferenceType=null, hasPrimary=false ] {org.apache.stratos.autoscaler.monitor.ClusterMonitor} [Error: null pointer: autoscalePolicy.getLoadThresholds()] [Near : {... autoscalePolicy.getLoadThresho ....}] Regards Manas -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639<tel:%2B94777568639> | +94783498120<tel:%2B94783498120> -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639<tel:%2B94777568639> | +94783498120<tel:%2B94783498120> -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639<tel:%2B94777568639> | +94783498120<tel:%2B94783498120> -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639<tel:%2B94777568639> | +94783498120<tel:%2B94783498120> -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639<tel:%2B94777568639> | +94783498120<tel:%2B94783498120> -- Raj -- Raj
