+Imesh with this emaill chain The AC ( auto scaling ) still not work. Looks like the puppet agent do not send back the system/health check status. puppet agent test is good but not health status back to master.
testing result: root@phpphp301-7c5:~# puppet agent --noop --test Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for phpphp301-7c5.prodstage.com Info: Applying configuration version '1425798573' Notice: /Stage[main]/Java/Exec[Install java]/returns: current_value notrun, should be 0 (noop) Notice: Class[Java]: Would have triggered 'refresh' from 1 events Notice: /Stage[main]/Php/Exec[update-apt]/returns: current_value notrun, should be 0 (noop) Notice: /Stage[main]/Php/Exec[enable ssl module]/returns: current_value notrun, should be 0 (noop) Notice: /Stage[main]/Php/Exec[enable ssl]/returns: current_value notrun, should be 0 (noop) Notice: /Stage[main]/Php/Exec[apache2 restart]/returns: current_value notrun, should be 0 (noop) Notice: Class[Php]: Would have triggered 'refresh' from 4 events Info: Class[Php]: Scheduling refresh of Class[Agent] Notice: Class[Agent]: Would have triggered 'refresh' from 1 events Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[extensions/artifacts-updated.sh] Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[extensions/start-servers.sh] Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[extensions/instance-activated.sh] Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[extensions/instance-started.sh] Info: Class[Agent]: Scheduling refresh of Agent::Start[cartridge-agent] Info: Class[Agent]: Scheduling refresh of Agent::Initialize[cartridge-agent] Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[conf/templates/jndi.properties.template] Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[bin/stratos.sh] Info: Class[Agent]: Scheduling refresh of Exec[copy launch-params to carbon_home] Info: Class[Agent]: Scheduling refresh of Agent::Push_templates[extensions/clean.sh] Notice: Agent::Initialize[cartridge-agent]: Would have triggered 'refresh' from 1 events Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of Exec[creating_target_for_cartridge-agent] Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of Exec[extracting_stratoscartridge-agent-4.0.0.zip_for_cartridge-agent] Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of Exec[downloading_apache-stratos-cartridge-agent-4.0.0.zip] Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of Exec[setting_permission_for_cartridge-agent] Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of Exec[creating_local_package_repo_for_cartridge-agent] Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[creating_local_package_repo_for_cartridge-agent]: Would have triggered 'refresh' from 1 events Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[creating_target_for_cartridge-agent]/returns: current_value notrun, should be 0 (noop) Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[creating_target_for_cartridge-agent]: Would have triggered 'refresh' from 1 events Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[downloading_apache-stratos-cartridge-agent-4.0.0.zip]: Would have triggered 'refresh' from 1 events Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[extracting_stratoscartridge-agent-4.0.0.zip_for_cartridge-agent]: Would have triggered 'refresh' from 1 events Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[setting_permission_for_cartridge-agent]/returns: current_value notrun, should be 0 (noop) Notice: /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[setting_permission_for_cartridge-agent]: Would have triggered 'refresh' from 1 events Notice: Agent::Initialize[cartridge-agent]: Would have triggered 'refresh' from 7 events Notice: Agent::Push_templates[bin/stratos.sh]: Would have triggered 'refresh' from 1 events Notice: Agent::Push_templates[extensions/instance-activated.sh]: Would have triggered 'refresh' from 1 events Notice: /Stage[main]/Agent/Exec[copy launch-params to carbon_home]/returns: current_value notrun, should be 0 (noop) Notice: /Stage[main]/Agent/Exec[copy launch-params to carbon_home]: Would have triggered 'refresh' from 1 events Notice: Agent::Push_templates[extensions/clean.sh]: Would have triggered 'refresh' from 1 events Notice: Agent::Push_templates[conf/templates/jndi.properties.template]: Would have triggered 'refresh' from 1 events Notice: Agent::Push_templates[extensions/start-servers.sh]: Would have triggered 'refresh' from 1 events Notice: Agent::Push_templates[extensions/instance-started.sh]: Would have triggered 'refresh' from 1 events Notice: Agent::Push_templates[extensions/artifacts-updated.sh]: Would have triggered 'refresh' from 1 events Notice: Agent::Start[cartridge-agent]: Would have triggered 'refresh' from 1 events Info: Agent::Start[cartridge-agent]: Scheduling refresh of Exec[starting_cartridge-agent] Notice: /Stage[main]/Agent/Agent::Start[cartridge-agent]/Exec[starting_cartridge-agent]: Would have triggered 'refresh' from 1 events Notice: Agent::Start[cartridge-agent]: Would have triggered 'refresh' from 1 events Notice: Class[Agent]: Would have triggered 'refresh' from 4 events Notice: Stage[main]: Would have triggered 'refresh' from 3 events Notice: Finished catalog run in 3.41 seconds Please advice, Thanks -D On Sun, Mar 8, 2015 at 12:09 AM, david hbase <davidhb...@gmail.com> wrote: > After recreate the CA certificate and the puppet master certificate, the > above error has been fixed. > But the AC ( auto scaling ) still not work. Looks like the puppet agent do > not send back the system/health check status. > > Any advice? > > Thanks > -D > > > On Sat, Mar 7, 2015 at 8:09 PM, david hbase <davidhb...@gmail.com> wrote: > >> Got another error with autoscaling, the puppet log as following: (Stratos >> 4.0 ) >> >> [2015-03-08 04:02:01] - -> >> /production/file_metadatas/modules/agent/activemq?links=manage&recurse=true&checksum_type=md5 >> [2015-03-08 04:02:02] 10.203.50.169 - - [08/Mar/2015:04:02:01 UTC] "PUT >> /production/report/phpphp201-25e.prodstage.com HTTP/1.1" 200 9 >> [2015-03-08 04:02:02] - -> /production/report/phpphp201-25e.prodstage.com >> [2015-03-08 04:02:49] ERROR OpenSSL::SSL::SSLError: SSL_accept SYSCALL >> returned=5 errno=0 state=unknown state >> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:32:in >> `accept' >> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:32:in >> `listen' >> /usr/lib/ruby/1.8/webrick/server.rb:173:in `call' >> /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread' >> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start' >> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread' >> /usr/lib/ruby/1.8/webrick/server.rb:95:in `start' >> /usr/lib/ruby/1.8/webrick/server.rb:92:in `each' >> /usr/lib/ruby/1.8/webrick/server.rb:92:in `start' >> /usr/lib/ruby/1.8/webrick/server.rb:23:in `start' >> /usr/lib/ruby/1.8/webrick/server.rb:82:in `start' >> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:27:in >> `listen' >> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:26:in >> `initialize' >> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:26:in `new' >> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:26:in >> `listen' >> /usr/lib/ruby/vendor_ruby/puppet/network/server.rb:27:in `start' >> /usr/lib/ruby/vendor_ruby/puppet/daemon.rb:139:in `start' >> /usr/lib/ruby/vendor_ruby/puppet/application/master.rb:286:in >> `start_webrick_master' >> /usr/lib/ruby/vendor_ruby/puppet/application/master.rb:199:in `main' >> /usr/lib/ruby/vendor_ruby/puppet/application/master.rb:160:in >> `run_command' >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook' >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >> /usr/lib/ruby/vendor_ruby/puppet/util.rb:488:in `exit_on_fail' >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run' >> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute' >> /usr/bin/puppet:8 >> [2015-03-08 04:02:49] ERROR OpenSSL::SSL::SSLError: SSL_accept SYSCALL >> returned=5 errno=0 state=unknown state >> >> Also, All the instance are auto expired after about 10 minutes and the >> new instances created automatically. >> >> Thanks >> -D >> >> On Fri, Mar 6, 2015 at 5:22 PM, Udara Liyanage <ud...@wso2.com> wrote: >> >>> Great to hear David >>> On 7 Mar 2015 06:42, "david hbase" <davidhb...@gmail.com> wrote: >>> >>>> Thanks you guys, The issue has been solved, following the 646 >>>> instruction. >>>> >>>> -D >>>> >>>> On Fri, Mar 6, 2015 at 3:12 PM, david hbase <davidhb...@gmail.com> >>>> wrote: >>>> >>>>> Great! It's working NOW!!! >>>>> >>>>> Just additional issue, instance can spawn, but no IP address back, >>>>> Logs as following ( with openstack deployement) >>>>> >>>>> [2015-03-06 22:53:10,437] INFO >>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>> - Trying to spawn an instance via cloud controller: [cluster] >>>>> php.php.domain [partition] zone-1 [lb-cluster] null [network-partition-id] >>>>> openstack-region1 >>>>> [2015-03-06 22:53:25,542] INFO >>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>> Instance is successfully starting up. MemberContext >>>>> [memberId=php.php.domainee0dea8b-a7c5-4c42-ba56-45305ade31a3, >>>>> nodeId=us-internal-1/20971aad-2737-48ac-96f7-7f318dc0a908, >>>>> clusterId=php.php.domain, cartridgeType=php, privateIpAddress=null, >>>>> publicIpAddress=null, allocatedIpAddress=null, initTime=1425682390438, >>>>> lbClusterId=null, networkPartitionId=openstack-region1] >>>>> [2015-03-06 22:53:25,930] ERROR >>>>> {org.apache.stratos.cloud.controller.iaases.OpenstackNovaIaas} - Failed >>>>> to >>>>> allocate an IP address. >>>>> [2015-03-06 22:53:25,931] ERROR >>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>> Error occurred while allocating an ip address. MemberContext >>>>> [memberId=php.php.domainee0dea8b-a7c5-4c42-ba56-45305ade31a3, >>>>> nodeId=us-internal-1/20971aad-2737-48ac-96f7-7f318dc0a908, >>>>> clusterId=php.php.domain, cartridgeType=php, privateIpAddress=null, >>>>> publicIpAddress=null, allocatedIpAddress=null, initTime=1425682390438, >>>>> lbClusterId=null, networkPartitionId=openstack-region1] >>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>> Failed to allocate an IP address. >>>>> at >>>>> org.apache.stratos.cloud.controller.iaases.OpenstackNovaIaas.associateAddress(OpenstackNovaIaas.java:259) >>>>> at >>>>> org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl$IpAllocator.run(CloudControllerServiceImpl.java:635) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Exception in thread "pool-31-thread-1" >>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>> Error occurred while allocating an ip address. MemberContext >>>>> [memberId=php.php.domainee0dea8b-a7c5-4c42-ba56-45305ade31a3, >>>>> nodeId=us-internal-1/20971aad-2737-48ac-96f7-7f318dc0a908, >>>>> clusterId=php.php.domain, cartridgeType=php, privateIpAddress=null, >>>>> publicIpAddress=null, allocatedIpAddress=null, initTime=1425682390438, >>>>> lbClusterId=null, networkPartitionId=openstack-region1] >>>>> at >>>>> org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl$IpAllocator.run(CloudControllerServiceImpl.java:694) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: >>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>> Failed to allocate an IP address. >>>>> at >>>>> org.apache.stratos.cloud.controller.iaases.OpenstackNovaIaas.associateAddress(OpenstackNovaIaas.java:259) >>>>> at >>>>> org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl$IpAllocator.run(CloudControllerServiceImpl.java:635) >>>>> ... 3 more >>>>> >>>>> Just following the https://issues.apache.org/jira/browse/STRATOS-646 >>>>> instruction, but still not work. >>>>> >>>>> Please help. >>>>> Thanks >>>>> -D >>>>> >>>>> >>>>> On Wed, Mar 4, 2015 at 10:41 PM, Udara Liyanage <ud...@wso2.com> >>>>> wrote: >>>>> >>>>>> >>>>>> On 5 Mar 2015 10:15, "david hbase" <davidhb...@gmail.com> wrote: >>>>>> > >>>>>> > Great appreciated your help. The json code has been attached: >>>>>> > >>>>>> > { >>>>>> > "id": "zone-1", >>>>>> > "provider": "openstack", >>>>>> > "property": [ >>>>>> > { >>>>>> > "name": "region", >>>>>> > "value": "us-internal-1" >>>>>> > } >>>>>> > ] >>>>>> > } >>>>>> > >>>>>> > >>>>>> > { >>>>>> > "type": "php", >>>>>> > "provider": "apache", >>>>>> > "host": "cartridge-php", >>>>>> > "displayName": "PHP", >>>>>> > "description": "PHP Cartridge", >>>>>> > "version": "8", >>>>>> > "multiTenant": true, >>>>>> > "persistance": true, >>>>>> > "portMapping": [ >>>>>> > { >>>>>> > "protocol": "http", >>>>>> > "port": "80", >>>>>> > "proxyPort": "8280" >>>>>> > } >>>>>> > ], >>>>>> > "deployment": {}, >>>>>> > "iaasProvider": [ >>>>>> > { >>>>>> > "type": "openstack", >>>>>> > "imageId": >>>>>> "us-internal-1/287614db-0033-4b9a-80ae-7f5e6be0a942", >>>>>> > "property": [ >>>>>> > { >>>>>> > "name": "instanceType", >>>>>> > "value": >>>>>> "us-internal-1/c33143d9-ce0a-472c-944b-3e39476970d3" >>>>>> > } >>>>>> > ] >>>>>> > } >>>>>> > ] >>>>>> > } >>>>>> > >>>>>> > { >>>>>> > "serviceName": "appserver", >>>>>> > "cartridgeType": "php", >>>>>> > "deploymentPolicyName": "economyDeployment", >>>>>> > "autoscalingPolicyName": "economyPolicy", >>>>>> > "clusterDomain": "stratos40build05", >>>>>> > "clusterSubDomain": "php", >>>>>> > "tenantRange": "*" >>>>>> > } >>>>>> > >>>>>> > [2015-03-05 01:58:39,126] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:58:39,129] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment >>>>>> > [2015-03-05 01:58:39,923] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:59:03,234] ERROR >>>>>> {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - No >>>>>> active instances are found for cartridge [php] >>>>>> >>>>>> This error message is shown when you try to subscribe to php >>>>>> cartridge, but there is no php instance which has become active. Could >>>>>> you >>>>>> please check weather the php instance became active. You can check >>>>>> cartridge agents log. >>>>>> >>>>>> > [2015-03-05 01:59:08,566] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment2 >>>>>> > [2015-03-05 01:59:11,445] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment2 >>>>>> > [2015-03-05 01:59:13,100] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment2 >>>>>> > [2015-03-05 01:59:14,574] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:59:14,577] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment >>>>>> > [2015-03-05 01:59:14,628] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:59:14,631] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment >>>>>> > [2015-03-05 01:59:15,582] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:59:15,585] INFO >>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>> economyDeployment >>>>>> > [2015-03-05 01:59:15,664] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:59:16,549] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > [2015-03-05 01:59:16,626] INFO >>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>> All partitions [zone-1] were validated successfully, against the >>>>>> Cartridge: >>>>>> php >>>>>> > >>>>>> > Thanks >>>>>> > -D >>>>>> > >>>>>> > On Wed, Mar 4, 2015 at 6:13 PM, Udara Liyanage <ud...@wso2.com> >>>>>> wrote: >>>>>> >> >>>>>> >> Hi, >>>>>> >> >>>>>> >> First of all php is normally not considered a multi tenant >>>>>> service. However could you share your php cartridge json and service json >>>>>> which threw the exception. >>>>>> >> >>>>>> >> On 5 Mar 2015 05:21, "david hbase" <davidhb...@gmail.com> wrote: >>>>>> >>> >>>>>> >>> Hi team >>>>>> >>> >>>>>> >>> With Stratos 4.0 and Openstack, I have deployed all the steps >>>>>> successfully, only except with the last step for the Multi-Tenant Service >>>>>> failed, and the error log as following: >>>>>> >>> >>>>>> >>> Failed to deploy the Service [Cartridge type] php [alias] >>>>>> a92a1da2-57bf-4e86-89a3-146900993793 . Cause: null >>>>>> >>> >>>>>> >>> Please Advice, waiting for your reply. >>>>>> >>> >>>>>> >>> Thanks >>>>>> >>> -D >>>>>> >>> >>>>>> >>> >>>>>> >>> [2015-03-04 23:22:57,413] INFO >>>>>> {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Deploying >>>>>> service.. >>>>>> >>> [2015-03-04 23:22:57,434] INFO >>>>>> {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} >>>>>> - Generated key : AYhQR4NuUnTFVtCI >>>>>> >>> [2015-03-04 23:22:57,436] ERROR >>>>>> {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Failed to >>>>>> deploy the Service [Cartridge type] php [alias] >>>>>> 0405644c-c371-4201-a05c-b24c337d7896 . Cause: null >>>>>> >>> java.lang.NullPointerException >>>>>> >>> at >>>>>> org.apache.stratos.manager.deploy.service.ServiceDeploymentManager.deployService(ServiceDeploymentManager.java:322) >>>>>> >>> at >>>>>> org.apache.stratos.rest.endpoint.services.ServiceUtils.deployService(ServiceUtils.java:1133) >>>>>> >>> at >>>>>> org.apache.stratos.rest.endpoint.services.StratosAdmin.deployService(StratosAdmin.java:908) >>>>>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> >>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>> >>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> >>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> >>> at >>>>>> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) >>>>>> >>> at >>>>>> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) >>>>>> >>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194) >>>>>> >>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100) >>>>>> >>> at >>>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) >>>>>> >>> at >>>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) >>>>>> >>> at >>>>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) >>>>>> >>> at >>>>>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) >>>>>> >>> at >>>>>> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206) >>>>>> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>>>>> >>> at >>>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262) >>>>>> >>> at >>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>>>>> >>> at >>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>>>>> >>> at >>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>>>>> >>> at >>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>>>>> >>> at >>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>>>>> >>> at >>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>>>>> >>> at >>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>>>>> >>> at >>>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>>>>> >>> at >>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>>>>> >>> at >>>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>>>>> >>> at >>>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>>>>> >>> at >>>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>>>>> >>> at >>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>>>>> >>> at >>>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>>>>> >>> at >>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>>>>> >>> at >>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>>>>> >>> at >>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >>>>>> >>> at >>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) >>>>>> >>> at >>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >>>>>> >>> at >>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) >>>>>> >>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> >>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> >>> at java.lang.Thread.run(Thread.java:745) >>>>>> >>> >>>>>> > >>>>>> >>>>>> >>>>> >>>> >> >