Hi, Team:
I have fixed all error, but LB still not send out the LB LB statistics to
Stratos:
1. Stratos logs:
[2015-03-10 07:46:24,855] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will
not run since the LB statistics have not received before this cycle for
network partition openstack-region1
[2015-03-10 07:46:53,190] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Average Rif event: [cluster] lb.lb.domain [network-partition] zone-1
[value] 0.0
[2015-03-10 07:46:53,190] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Network partition context is not available for : [network partition]
zone-1
[2015-03-10 07:46:53,199] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Gradient of Rif event: [cluster] lb.lb.domain [network-partition] zone-1
[value] 0.0
[2015-03-10 07:46:53,199] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Network partition context is not available for : [network partition]
zone-1
[2015-03-10 07:46:53,393] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Second derivative of Rif event: [cluster] lb.lb.domain
[network-partition] zone-1 [value] 0.0
[2015-03-10 07:46:53,394] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Network partition context is not available for : [network partition]
zone-1
[2015-03-10 07:46:54,855] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor
is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb,
deploymentPolicy=Deployment Policy [id]economyDeployment [partitions]
[org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@274ba6a4],
autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null,
description=null], lbReferenceType=null]
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum
check for partition zone-1
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
[network-partition] openstack-region1 [partition] zone-1
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
[network-partition] openstack-region1 [partition] zone-1 Non terminated
member count: 1
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
[network-partition] openstack-region1 [partition] zone-1 Minimum member
count: 1
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances
rule: [partition] zone-1 [network-partition] openstack-region1
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
[network-partition] openstack-region1 [partition] zone-1 Obsolete member
count: 0
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
check executed for : org.apache.stratos.autoscaler.PartitionContext@6bf159e8
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will
not run since the LB statistics have not received before this cycle for
network partition openstack-region1
[2015-03-10 07:47:24,857] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor
is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb,
deploymentPolicy=Deployment Policy [id]economyDeployment [partitions]
[org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@274ba6a4],
autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null,
description=null], lbReferenceType=null]
[2015-03-10 07:47:24,858] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum
check for partition zone-1
[2015-03-10 07:47:24,858] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
[network-partition] openstack-region1 [partition] zone-1
[2015-03-10 07:47:24,858] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
[network-partition] openstack-region1 [partition] zone-1 Non terminated
member count: 1
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
[network-partition] openstack-region1 [partition] zone-1 Minimum member
count: 1
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances
rule: [partition] zone-1 [network-partition] openstack-region1
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
[network-partition] openstack-region1 [partition] zone-1 Obsolete member
count: 0
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
check executed for : org.apache.stratos.autoscaler.PartitionContext@6bf159e8
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will
not run since the LB statistics have not received before this cycle for
network partition openstack-region1
2. LB logs:
root@lblbdoma-59:~# /mnt/apache-stratos-load-balancer-4.0.0/bin/stratos.sh
JAVA_HOME environment variable is set to /usr/lib/jvm/java-7-openjdk-amd64
CARBON_HOME environment variable is set to
/mnt/apache-stratos-load-balancer-4.0.0
[2015-03-10 07:38:38,577] INFO
{org.wso2.carbon.server.extensions.PatchInstaller} - Patch changes
detected
[2015-03-10 07:38:40,233] INFO
{org.wso2.carbon.server.util.PatchUtils.console} - Backed up plugins to
patch0000
[2015-03-10 07:38:41,537] INFO
{org.wso2.carbon.server.util.PatchUtils.console} - Patch verification
started
[2015-03-10 07:38:41,831] INFO
{org.wso2.carbon.server.util.PatchUtils.console} - Patch verification
successfully completed without encountering any issues.
[2015-03-10 07:38:46,194] INFO - CarbonCoreActivator Starting WSO2
Carbon...
[2015-03-10 07:38:46,196] INFO - CarbonCoreActivator Operating System :
Linux 3.2.0-74-virtual, amd64
[2015-03-10 07:38:46,196] INFO - CarbonCoreActivator Java Home :
/usr/lib/jvm/java-7-openjdk-amd64/jre
[2015-03-10 07:38:46,197] INFO - CarbonCoreActivator Java Version :
1.7.0_75
[2015-03-10 07:38:46,197] INFO - CarbonCoreActivator Java VM :
OpenJDK 64-Bit Server VM 24.75-b04,Oracle Corporation
[2015-03-10 07:38:46,197] INFO - CarbonCoreActivator Carbon Home :
/mnt/apache-stratos-load-balancer-4.0.0
[2015-03-10 07:38:46,197] INFO - CarbonCoreActivator Java Temp Dir :
/mnt/apache-stratos-load-balancer-4.0.0/tmp
[2015-03-10 07:38:46,197] INFO - CarbonCoreActivator User :
root, en-US, Etc/UTC
[2015-03-10 07:38:46,349] WARN - SystemValidator Could not validate the
system for configuration parameter : CPU
[2015-03-10 07:38:46,351] WARN - ValidationResultPrinter RAM size (MB):
2003 of the system is below the recommended minimum size :2048
[2015-03-10 07:38:46,357] WARN - ValidationResultPrinter The default
keystore (wso2carbon.jks) is currently being used. To maximize security
when deploying to a production environment, configure a new keystore with a
unique password in the production server profile.
[2015-03-10 07:38:46,357] WARN - ValidationResultPrinter Swap Memory size
(MB): 0 of the system is below the recommended minimum size :2048
[2015-03-10 07:38:46,366] INFO - AgentHolder Agent created !
[2015-03-10 07:38:46,411] INFO - AgentDS Successfully deployed Agent Client
[2015-03-10 07:39:33,357] INFO - EmbeddedRegistryService Configured
Registry in 225ms
[2015-03-10 07:39:33,847] INFO - RegistryCoreServiceComponent Registry
Mode : READ-WRITE
[2015-03-10 07:39:34,793] INFO - UserStoreMgtDSComponent Carbon
UserStoreMgtDSComponent activated successfully.
[2015-03-10 07:39:34,972] INFO - ClusterBuilder Clustering has been
disabled
[2015-03-10 07:39:35,273] INFO - CarbonAxisConfigurator No modules
directory was found under
/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/server.
[2015-03-10 07:39:35,326] INFO - PassThroughHttpSSLSender Initializing
Pass-through HTTP/S Sender...
[2015-03-10 07:39:35,344] INFO - ClientConnFactoryBuilder HTTPS Loading
Identity Keystore from : repository/resources/security/wso2carbon.jks
[2015-03-10 07:39:35,349] INFO - ClientConnFactoryBuilder HTTPS Loading
Trust Keystore from : repository/resources/security/client-truststore.jks
[2015-03-10 07:39:35,380] INFO - PassThroughHttpSSLSender Pass-through
HTTPS Sender started...
[2015-03-10 07:39:35,380] INFO - PassThroughHttpSender Initializing
Pass-through HTTP/S Sender...
[2015-03-10 07:39:35,381] INFO - PassThroughHttpSender Pass-through HTTP
Sender started...
[2015-03-10 07:39:35,398] INFO - PassThroughHttpSSLListener Initializing
Pass-through HTTP/S Listener...
[2015-03-10 07:39:35,415] INFO - PassThroughHttpListener Initializing
Pass-through HTTP/S Listener...
[2015-03-10 07:39:35,501] INFO - ModuleDeployer Deploying module:
addressing-1.6.1-wso2v10 -
file:/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/client/modules/addressing-1.6.1-wso2v10.mar
[2015-03-10 07:39:35,508] INFO - ModuleDeployer Deploying module:
rampart-1.6.1-wso2v1 -
file:/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/client/modules/rampart-1.6.1-wso2v1.mar
[2015-03-10 07:39:35,513] INFO - TCPTransportSender TCP Sender started
[2015-03-10 07:39:37,032] INFO - CarbonServerManager Repository :
/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/server/
[2015-03-10 07:39:37,195] INFO - PermissionUpdater Permission cache
updated for tenant -1234
[2015-03-10 07:39:37,297] INFO - ServiceBusInitializer Starting ESB...
[2015-03-10 07:39:37,335] INFO - ServiceBusInitializer Initializing Apache
Synapse...
[2015-03-10 07:39:37,346] INFO - SynapseControllerFactory Using Synapse
home : /mnt/apache-stratos-load-balancer-4.0.0/.
[2015-03-10 07:39:37,353] INFO - SynapseControllerFactory Using
synapse.xml location :
/mnt/apache-stratos-load-balancer-4.0.0/././repository/deployment/server/synapse-configs/default
[2015-03-10 07:39:37,353] INFO - SynapseControllerFactory Using server
name : localhost
[2015-03-10 07:39:37,360] INFO - SynapseControllerFactory The timeout
handler will run every : 15s
[2015-03-10 07:39:37,371] INFO - Axis2SynapseController Initializing
Synapse at : Tue Mar 10 07:39:37 UTC 2015
[2015-03-10 07:39:37,377] INFO - CarbonSynapseController Loading the
mediation configuration from the file system
[2015-03-10 07:39:37,379] INFO - MultiXMLConfigurationBuilder Building
synapse configuration from the synapse artifact repository at :
././repository/deployment/server/synapse-configs/default
[2015-03-10 07:39:37,385] INFO - XMLConfigurationBuilder Generating the
Synapse configuration model by parsing the XML configuration
[2015-03-10 07:39:37,732] INFO - SynapseConfigurationBuilder Loaded
Synapse configuration from the artifact repository at :
././repository/deployment/server/synapse-configs/default
[2015-03-10 07:39:37,735] INFO - Axis2SynapseController Loading mediator
extensions...
[2015-03-10 07:39:37,736] INFO - Axis2SynapseController Deploying the
Synapse service...
[2015-03-10 07:39:37,737] INFO - Axis2SynapseController Deploying Proxy
services...
[2015-03-10 07:39:37,737] INFO - Axis2SynapseController Deploying
EventSources...
[2015-03-10 07:39:37,762] INFO - ServerManager Server ready for
processing...
[2015-03-10 07:39:37,850] WARN - LoadBalancerConfiguration Endpoint
timeout not found, using default: 15000
[2015-03-10 07:39:37,852] INFO - SynapseConfigurator Configuring synapse
main sequence...
[2015-03-10 07:39:37,938] INFO - SynapseConfigurator Synapse main sequence
configured successfully
[2015-03-10 07:39:37,948] INFO - LoadBalancerServiceComponent Tenant
receiver thread started
[2015-03-10 07:39:37,967] INFO - LoadBalancerServiceComponent Topology
receiver thread started
[2015-03-10 07:39:37,967] INFO - TenantEventMessageDelegator Tenant event
message delegator started
[2015-03-10 07:39:37,974] INFO - LoadBalancerTopologyEventReceiver Load
balancer topology receiver thread started
[2015-03-10 07:39:38,060] INFO - TopologyEventMessageDelegator Topology
event message delegator started
[2015-03-10 07:39:38,183] INFO - LoadBalancerServiceComponent Load
balancer statistics notifier thread started
[2015-03-10 07:39:38,189] INFO - LoadBalancerServiceComponent Load
balancer service component is activated
[2015-03-10 07:39:38,677] INFO - PassThroughHttpSSLListener Starting
Pass-through HTTPS Listener...
[2015-03-10 07:39:38,701] INFO - PassThroughHttpSSLListener Pass-through
HTTPS Listener started on 0:0:0:0:0:0:0:0:443
[2015-03-10 07:39:38,701] INFO - PassThroughHttpListener Starting
Pass-through HTTP Listener...
[2015-03-10 07:39:38,709] INFO - PassThroughHttpListener Pass-through HTTP
Listener started on 0:0:0:0:0:0:0:0:80
[2015-03-10 07:39:38,915] INFO - TaskServiceImpl Task service starting in
STANDALONE mode...
[2015-03-10 07:39:39,026] INFO - JMXServerManager JMX Service URL :
service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2015-03-10 07:39:39,026] INFO - StartupFinalizerServiceComponent
Server : Apache Stratos LB-4.0.0
[2015-03-10 07:39:39,027] INFO - StartupFinalizerServiceComponent WSO2
Carbon started in 60 sec
[2015-03-10 07:39:39,317] INFO - CarbonUIServiceComponent Mgt Console URL
: https://10.203.50.158:9443/carbon/
[2015-03-10 07:40:01,081] INFO - CompleteTenantMessageProcessor Tenant
initialized
[2015-03-10 07:40:01,093] INFO - CompleteTopologyMessageProcessor Topology
initialized
Please help, Thanks
-D
On Mon, Mar 9, 2015 at 10:02 PM, david hbase <[email protected]> wrote:
> Hi, Team:
>
> I found the ERROR with Stratos 4.0 loadbalancer as following, and this
> error blocked the LB statistics to CEP,
>
> Please help, Thanks
> -D
>
> [2015-03-10 04:59:46,057] ERROR - TopicSubscriber Error while subscribing
> to the topic: topology
> javax.naming.NoInitialContextException: Failed to create InitialContext
> using factory specified in hash table. [Root exception is
> java.lang.ClassNotFoundException: class
> org.apache.activemq.jndi.ActiveMQInitialContextFactory not found]
> at
> org.wso2.carbon.context.internal.CarbonContextDataHolder$CarbonInitialJNDIContextFactoryBuilder.createInitialContextFactory(CarbonContextDataHolder.java:737)
> at
> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:681)
> at
> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
> at javax.naming.InitialContext.init(InitialContext.java:242)
> at javax.naming.InitialContext.<init>(InitialContext.java:216)
> at
> org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:59)
> at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:63)
> at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:116)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: class
> org.apache.activemq.jndi.ActiveMQInitialContextFactory not found
> at
> org.wso2.carbon.context.internal.CarbonContextDataHolder.classForName(CarbonContextDataHolder.java:1450)
> at
> org.wso2.carbon.context.internal.CarbonContextDataHolder.access$700(CarbonContextDataHolder.java:103)
> at
> org.wso2.carbon.context.internal.CarbonContextDataHolder$CarbonInitialJNDIContextFactoryBuilder.createInitialContextFactory(CarbonContextDataHolder.java:690)
> ... 8 more
> [2015-03-10 04:59:46,060] INFO - TopicSubscriber Will try to subscribe
> again in 30 sec
> [2015-03-10 04:59:46,061] INFO - TopicSubscriber Will try to subscribe
> again in 30 sec
>
> On Mon, Mar 9, 2015 at 7:36 AM, Rajkumar Rajaratnam <[email protected]>
> wrote:
>
>> Hi David,
>>
>> You can set the member expiry timeout in
>> Stratos-Home/repository/conf/autoscaler.xml via expiryTimeout element
>> (Highlighted below)
>>
>> <configuration>
>> <autoscaler>
>> <rulesEvaluator>
>> <schedule>
>> <initialDelay>30</initialDelay>
>> <period>15</period>
>> </schedule>
>> </rulesEvaluator>
>> <cloudController>
>> <hostname>localhost</hostname>
>> <port>9444</port>
>> <!-->CC client timout in ms<-->
>> <clientTimeout>300000</clientTimeout>
>> </cloudController>
>> <stratosManager>
>> <hostname>localhost</hostname>
>> <port>9445</port>
>> <!--SM client timeout in ms-->
>> <clientTimeout>300000</clientTimeout>
>> </stratosManager>
>> <member>
>> <!-->member expiry timeout (time waited till activation) in
>> ms<-->
>> *<expiryTimeout>900000</expiryTimeout>*
>> </member>
>> <!-- monitor task interval -->
>> <monitorInterval>90000</monitorInterval>
>> </autoscaler>
>> </configuration>
>>
>> Thanks.
>>
>>
>> On Mon, Mar 9, 2015 at 7:56 PM, david hbase <[email protected]> wrote:
>>
>>> Hi, Imesh and Lakmal
>>>
>>> Thanks for you support, I will get more detail debug logs.
>>> An further issue is how can we disable / change the Member expriy time?
>>>
>>> [2015-03-08 09:41:10,723] DEBUG
>>> {org.apache.stratos.autoscaler.PartitionContext} - Member expiry time is
>>> set to: 900000
>>>
>>> Thanks
>>> -D
>>>
>>> On Mon, Mar 9, 2015 at 2:35 AM, Lakmal Warusawithana <[email protected]>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Mar 9, 2015 at 2:39 PM, Imesh Gunaratne <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi Lakmal,
>>>>>
>>>>> On Mon, Mar 9, 2015 at 12:20 PM, Lakmal Warusawithana <[email protected]
>>>>> > wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 9, 2015 at 12:16 PM, Imesh Gunaratne <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> We might need to check why the load balancer is not sending
>>>>>>> statistics to CEP. Try to enable debug logs in the load balancer and
>>>>>>> run,
>>>>>>> it should trace the statistics sent to CEP.
>>>>>>>
>>>>>>> No AFAIK, it is not possible to configure Autoscaler to disregard
>>>>>>> load balancer statistics. I see this as a very valid requirement (to
>>>>>>> exclude any of the scaling factors), may be we can consider adding this
>>>>>>> to
>>>>>>> 4.1.0. Would you mind reporting this as a feature request?
>>>>>>>
>>>>>>>
>>>>>> I'm bit confuse here! I can scale up a cluster without having a LB.
>>>>>> (in 4.1.0) which mean LB is not a MUST right?
>>>>>>
>>>>>
>>>>> Yes, scaling up would work but scaling down might not work if
>>>>> statistics for all the autoscaling factors are available:
>>>>>
>>>>> scaleUp : Boolean() from ((rifReset && (rifPredictedValue >
>>>>> rifAverageLimit * 0.8)) || (mcReset && (mcPredictedValue > mcAverageLimit
>>>>> *
>>>>> 0.8)) || (laReset && (laPredictedValue > laAverageLimit * 0.8)))
>>>>> scaleDown : Boolean() from ((rifReset && (rifPredictedValue <
>>>>> rifAverageLimit * 0.1)) && (mcReset && (mcPredictedValue < mcAverageLimit
>>>>> *
>>>>> 0.1)) && (laReset && (laPredictedValue < laAverageLimit * 0.1)))
>>>>>
>>>>>
>>>>> https://github.com/apache/stratos/blob/4.0.0/products/stratos/modules/distribution/src/main/conf/scaling.drl
>>>>>
>>>>>
>>>> IMO, we should fix this. We should support scale up/down with selected
>>>> autoscaling parameters.
>>>>
>>>>
>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lakmal Warusawithana
>>>> Vice President, Apache Stratos
>>>> Director - Cloud Architecture; WSO2 Inc.
>>>> Mobile : +94714289692
>>>> Blog : http://lakmalsview.blogspot.com/
>>>>
>>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
