Hi Manas, I quickly went through the logs and mincheck.drl. And these are my observations.
- Nothing wrong with mincheck.drl - Cluster monitor is created successfully - Cluster monitor is executing monitor() method - monitor() executes mincheck drool, since min rule is not satisfied, it delegates request to RuleTaskDelegator to spawn an instance. - RuleTaskDelegator calls CC to create an instance which is failing[1] - So mincheck rule is failing, implies that monitor is failing So the root cause is instance creation is failing. 1 = > TID: [0] [STRATOS] [2014-11-10 07:24:01,648] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Trying to spawn an instance via cloud controller: [cluster] imc-dm.imc-dm.domain [partition] RegionOne-Core [lb-cluster] null [network-partition-id] N1 {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} TID: [0] [STRATOS] [2014-11-10 07:24:01,651] ERROR {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor: Monitor failed.ClusterMonitor [clusterId=imc-ce.imc-ce.domain, serviceId=imc-ce, deploymentPolicy=Deployment Policy [id]static-1-Core [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@5126e236], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null, hasPrimary=false ] {org.apache.stratos.autoscaler.monitor.ClusterMonitor} Exception executing consequence for rule "Minimum Rule" in org.apache.stratos.autoscaler.rule: [Error: $delegator.delegateSpawn($ctxt, clusterId, lbRef, isPrimary): Cannot spawn an instance] [Near : {... if (isPrimary){ ....}] ^ [Line: 1, Column: 1] at org.drools.runtime.rule.impl.DefaultConsequenceExceptionHandler.handleException(DefaultConsequenceExceptionHandler.java:39) at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1297) at org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221) at org.drools.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1456) at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:710) at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:674) at org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:230) at org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator.evaluateMinCheck(AutoscalerRuleEvaluator.java:94) at org.apache.stratos.autoscaler.monitor.ClusterMonitor.monitor(ClusterMonitor.java:157) at org.apache.stratos.autoscaler.monitor.ClusterMonitor.run(ClusterMonitor.java:86) at java.lang.Thread.run(Thread.java:745) Caused by: [Error: $delegator.delegateSpawn($ctxt, clusterId, lbRef, isPrimary): Cannot spawn an instance] [Near : {... if (isPrimary){ ....}] ^ [Line: 1, Column: 1] at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:416) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.optimizeAccessor(ReflectiveAccessorOptimizer.java:143) at org.mvel2.ast.ASTNode.optimize(ASTNode.java:159) at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:115) at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85) at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113) at org.mvel2.MVEL.executeExpression(MVEL.java:930) at org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104) at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287) ... 9 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1104) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:987) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:377) ... 19 more Caused by: java.lang.RuntimeException: Cannot spawn an instance at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegateSpawn(RuleTasksDelegator.java:107) ... 26 more Caused by: org.apache.stratos.autoscaler.exception.SpawningException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:174) at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegateSpawn(RuleTasksDelegator.java:87) ... 26 more Caused by: org.apache.axis2.AxisFault: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.fromOM(CloudControllerServiceStub.java:4312) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.startInstance(CloudControllerServiceStub.java:1415) at org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:162) ... 27 more Caused by: org.apache.axiom.om.impl.exception.OMStreamingException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at org.apache.axiom.om.impl.SwitchingWrapper.getElementText(SwitchingWrapper.java:966) at javax.xml.stream.util.StreamReaderDelegate.getElementText(StreamReaderDelegate.java:100) at org.apache.stratos.cloud.controller.stub.pojo.MemberContext$Factory.parse(MemberContext.java:1470) at org.apache.stratos.cloud.controller.stub.impl.StartInstanceResponse$Factory.parse(StartInstanceResponse.java:417) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.fromOM(CloudControllerServiceStub.java:4159) ... 29 more Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791] Message: elementGetText() function expects text only elment but START_ELEMENT was encountered. at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.getElementText(XMLStreamReaderImpl.java:856) at org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getElementText(XMLStreamReaderWrapper.java:100) at org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getElementText(XMLStreamReaderWrapper.java:100) at org.apache.axiom.om.impl.SwitchingWrapper.getElementText(SwitchingWrapper.java:962) We need to look at why instance creation is failing. Do you have any idea why instance creation is failing? I am looking into it further. Thanks. On Mon, Nov 10, 2014 at 3:31 PM, Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com> wrote: > Hi Raj, > > > > Mincheck.drl is attached herewith. > > > > Regards > > Manas > > > > *From:* Rajkumar Rajaratnam [mailto:rajkum...@wso2.com] > *Sent:* Monday, November 10, 2014 3:18 PM > > *To:* dev@stratos.apache.org > *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log > > > > Great Manas. > > I will go through the log and get back to you. It would be great if you > can add the mincheck.drl file too. > > Thanks. > > > > On Mon, Nov 10, 2014 at 3:10 PM, Manas Padegaonkar -T (mpadegao - ARICENT > TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com> wrote: > > Hi Raj, > > > > Attached is the wso2carbon.log as requested by you. > > > > Regards > > Manas > > > > *From:* Rajkumar Rajaratnam [mailto:rajkum...@wso2.com] > *Sent:* Monday, October 27, 2014 5:52 PM > > > *To:* dev@stratos.apache.org > *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log > > > > Will you be able to share the complete wso2carbon.log file? (at least from > the autoscale policy deployment upto this error message) > > > > Thanks. > > > > On Mon, Oct 27, 2014 at 5:41 PM, Manas Padegaonkar -T (mpadegao - ARICENT > TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com> wrote: > > Hi Raj, > > > > I am using the autoscale policy name correctly during subscription & > cartridge gets spawned, fetches catalog from puppet-master & becoming > Active. > > After this I am getting Cluster monitor: Monitor Failed. > > > > If I will use the wrong autoscale-policy name for cartridge subscription, > it will fail there only. > > > > Regards > > Manas > > > > *From:* Rajkumar Rajaratnam [mailto:rajkum...@wso2.com] > *Sent:* Monday, October 27, 2014 5:20 PM > > > *To:* dev@stratos.apache.org > *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log > > > > oops. so we don't need subscription json to subscribe. I confused with > kubernetes stuff. > > > > AFAIK, this error happens when the relevant autoscale policy is not found > in Policy Manager. When we pass autoscale policy name, AS is loading the > autoscale policy from Policy Manager. If the policy with the name you > provided is not there, it will give this error. Please make sure that you > have given the correct autoscale policy name. > > > > Thanks. > > > > On Mon, Oct 27, 2014 at 5:00 PM, Rajkumar Rajaratnam <rajkum...@wso2.com> > wrote: > > Hi Manas, > > > > If possible, can you share your subscription json and autoscale policy > json? > > > > Thanks. > > > > On Mon, Oct 27, 2014 at 4:56 PM, Manas Padegaonkar -T (mpadegao - ARICENT > TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com> wrote: > > Hi Raj, > > > > I am not using Stratos-GUI & working from CLI only. > > > > Regards > > Manas > > > > *From:* Rajkumar Rajaratnam [mailto:rajkum...@wso2.com] > *Sent:* Monday, October 27, 2014 4:53 PM > *To:* dev@stratos.apache.org > *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log > > > > I assumed you were using CLI or Curl commands. If you are using management > console, did you select the autoscale policy from autoscale policies drop > down list? And could you please tell me whether you are using master branch > or some other branch? > > > > Thanks. > > > > On Mon, Oct 27, 2014 at 4:42 PM, Rajkumar Rajaratnam <rajkum...@wso2.com> > wrote: > > Hi Manas, > > > > Reason is your autoscale policy is null. You need to give a deployed > autoscale policy name along with subscription json. > > > > Thanks. > > > > On Mon, Oct 27, 2014 at 4:36 PM, Manas Padegaonkar -T (mpadegao - ARICENT > TECHNOLOGIES HOLDINGS LIMITED at Cisco) <mpade...@cisco.com> wrote: > > Hi, > > > > I am getting following messages periodically in wso2carbon.log for all of > the subscribed cartridges which came Active. > > Want to know what is triggering this error to come & what is its impact if > any. > > > > > > > > TID: [0] [STRATOS] [2014-10-13 13:04:59,436] ERROR > {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor: > Monitor failed.ClusterMonitor [clusterId=1-14.1-14.domain, serviceId=1-14, > deploymentPolicy=Deployment Policy [id]static-1-Core [partitions] > [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@118c06f9], > autoscalePolicy=null, lbReferenceType=null, hasPrimary=false ] > {org.apache.stratos.autoscaler.monitor.ClusterMonitor} > > [Error: null pointer: autoscalePolicy.getLoadThresholds()] > > [Near : {... autoscalePolicy.getLoadThresho ....}] > > > > > > Regards > > Manas > > > > > > > > -- > > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > > Mobile +94777568639 | +94783498120 > > > > > > -- > > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > > Mobile +94777568639 | +94783498120 > > > > > > -- > > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > > Mobile +94777568639 | +94783498120 > > > > > > -- > > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > > Mobile +94777568639 | +94783498120 > > > > > > -- > > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > > Mobile +94777568639 | +94783498120 > > > > > -- > > Raj > -- Raj