Great work Akila. It is really nice to see that it will minimize the time taken to run the integration tests.
Thanks, Gayan On Mon, Nov 30, 2015 at 12:58 PM, Reka Thirunavukkarasu <r...@wso2.com> wrote: > Hi Akila > > It is really great work in identifying and optimizing lot of places in the > source code. We didn't review the complete flow earlier and improve any of > the time. This effort will definitely improve the overall performance and > reduce any intermittent concurrent issues that we faced with integration > tests sometimes. Nice to see that 20+ integration test is running in 15min. > > Thanks, > Reka > > On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera < > raviha...@wso2.com> wrote: > >> Hi, >> >> Currently there are lot of Thread.sleep calls in mock iaas component >> which makes it slow and cause unexpected behavior due to concurrency >> issues. Also it has a significant performance overhead when running >> integration tests since mock iaas is being used for test cases. I've been >> working on improving this component by doing following changes; >> >> - Remove *all* Thread sleep calls in mock iaas >> - Introduce a method named 'initialize' to start event receivers and >> publishers. This is a synchronous call which grantees that receiver and >> publisher objects will be created successfully. If not it will throw an >> exception and startInstance() method call in CC will fail. Earlier this >> task was delegated to an executor service which made it difficult to check >> whether mock instance was created successfully. >> - Create topology receiver in mock instance and listen for member >> initialized and member started events. It will publish instance started and >> instance activated events based on topology events received rather than >> sleeping for some time interval before publishing. >> >> After making those changes I faced multiple integration test failures. >> This was mainly because integration tests relied heavily on Thread sleep >> calls to assert various conditions. With these changes, the time taken for >> a mock instance/app to become active came down to milliseconds, hence test >> cases could not detect member status or app status correctly. Therefore I >> had to introduce a new non-blocking mechanism to check app/member status by >> using thread synchronization. >> >> Now the average time taken for complete integration tests is around 16 >> mins (earlier it was more than 30 mins). This is almost 50% performance >> gain. Created JIRA at [1]. >> >> >> Following is a summary of changes. >> >> AutoscalerTopologyEventReceiver: >> - Fix formatting and log messages >> - ClusterInstanceTerminatedEventListener check appMonitor is null when >> calling destroy() on monitor >> >> AutoscalerServiceImpl >> - Fix formatting and log messages >> >> ClusterStatusActiveProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> GroupStatusActiveProcessor >> - Fix formatting and log messages (log group-instance-id) >> - Print groups map entries and cluster data holder map entries if debug >> enabled >> >> GroupStatusProcessor >> - Fix formatting and log messages (log group-instance-id) >> >> GroupStatusTerminatedProcessor >> - Fix formatting and log messages (log group-instance-id) >> >> CloudControllerServiceComponent >> - Increase THREAD_POOL_SIZE from 10 to 20 >> >> >> TopologyBuilder >> - Fix formatting and log messages >> - Move acquire topology lock call outside of try block >> >> RestClient >> - Add logs for every method to help troubleshooting integration failures >> >> StratosTestServerManager >> - Call waitForPort method with restart timeout of 600000 ms. This is to >> avoid test failures due to slow builder machines. >> >> >> RestConstants >> - Add entity name: REPO_NOTIFY_NAME = "GitHook" >> >> TopologyHandler >> - Update timeout values: >> APPLICATION_ACTIVATION_TIMEOUT = 300000; >> APPLICATION_INACTIVATION_TIMEOUT = 120000; >> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000; >> MEMBER_TERMINATION_TIMEOUT = 120000; >> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000; >> >> - Increase executorService pool size from 10 to 30 to compensate for >> additional event receivers >> - Event receivers with logs for events >> - healthStatEventReceiver: MemberFaultEvent >> >> - applicationsEventReceiver: >> ApplicationInstanceActivatedEventListener, >> ApplicationInstanceInactivatedEventListener >> >> - topologyEventReceiver: MemberActivatedEventListener, >> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener, >> ClusterInstanceInactivateEventListener >> >> - Added logs for every method to help troubleshoot integration failures >> - Asynchronous mechanism to assertApplicationActiveStatus >> - Asynchronous mechanism to assertApplicationInactiveStatus >> >> >> Application >> - Added log in getStatus() method to print status of all application >> instances >> >> >> ApplicationCreatedMessageProcessor >> - Fix formatting and log messages >> >> ApplicationDeletedMessageProcessor >> - Fix formatting and log messages >> ApplicationInstanceActivatedMessageProcessor >> - Fix formatting and log messages (log app-instance-id) >> >> ApplicationInstanceCreatedMessageProcessor >> - Fix formatting and log messages (log app-instance-id) >> >> ApplicationInstanceInactivatedMessageProcessor >> - Fix formatting and log messages (log app-instance-id) >> >> ApplicationInstanceTerminatedMessageProcessor >> - Fix formatting and log messages (log app-instance-id) >> >> ApplicationInstanceTerminatingMessageProcessor >> - Fix formatting and log messages (log app-instance-id) >> >> ClusterStatusClusterActivatedMessageProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> ClusterStatusClusterInactivateMessageProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> ClusterStatusClusterInstanceCreatedMessageProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> ClusterStatusClusterResetMessageProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> ClusterStatusClusterTerminatedMessageProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> ClusterStatusClusterTerminatingMessageProcessor >> - Fix formatting and log messages (log cluster-instance-id) >> >> Introduce removeEventListener method for message processor chain to >> remove a registered event listener object. This is needed since integration >> tests will be registering event listeners on-demand. Those listeners needs >> to be removed at the end of the test case. >> - ApplicationSignUpMessageProcessorChain >> - ApplicationsMessageProcessorChain >> - ClusterStatusMessageProcessorChain >> - DomainMappingMessageProcessorChain >> - HealthStatMessageProcessorChain >> - InitializerMessageProcessorChain >> - InstanceNotifierMessageProcessorChain >> - InstanceStatusMessageProcessorChain >> - TenantMessageProcessorChain >> - TopologyMessageProcessorChain >> - MessageProcessorChain >> - ApplicationsEventMessageDelegator >> - ApplicationsEventReceiver >> >> InstanceNotifierEventReceiver >> - synchronized blocks for execute() and terminate() >> - eventSubscriber object creation moved to constructor from execute() >> method. This is to avoid possible NPE when calling terminate() method >> >> MetadataApi >> - Catch generic Exception instead of RegistryException >> >> DataStore >> - throws MetadataException in addition to RegistryException >> MetadataApiRegistry >> - throw new MetadataException instead of RegistryException >> >> >> MockIaasServiceComponent >> - Move mockIaasServiceUtil.startInstancesPersisted() to >> MockIaasServiceImpl() >> >> MockIaasServiceImpl >> - start persisted mock instances in the constructor >> - Remove all thread sleep calls >> >> >> MockIaasServiceUtil >> - Move startInstancesPersisted() method to MockIaasServiceImpl() >> >> >> MockInstance >> - Create TopologyEventReceiver object and register: >> MemberInitializedEventListener, MemberStartedEventListener, >> MemberMaintenanceListener >> - Publish InstanceStartedEvent upon receiving MemberInitializedEvent >> - Publish InstanceActivatedEvent upon receiving MemberStartedEvent >> - Publish InstanceReadyToShutdownEvent upon receiving >> MemberMaintenanceModeEvent >> - synchronized terminate() and initialize() methods >> - Introduce an initialize() method to start event receivers and health >> stat publisher >> - Introduce field 'memberStatus' of type MemberStatus to track the >> life-cycle of mock instance >> >> MockIaasServiceTest >> - Start embedded MB in unit test case since >> mockIaasService.startInstance call will start the event receivers as well >> >> GitHookTestCase >> - Make artifactUpdateEventCount an AtomicInteger >> - Replace restClient.doPost call with restClient.addEntity call >> - Terminate instanceNotifierEventReceiver at the end of test case >> - Shutdown eventListenerExecutorService at the end of test case >> >> [1] https://issues.apache.org/jira/browse/STRATOS-1633 >> >> Thanks. >> >> -- >> Akila Ravihansa Perera >> WSO2 Inc.; http://wso2.com/ >> >> Blog: http://ravihansa3000.blogspot.com >> > > > > -- > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > Mobile: +94776442007 > > > -- Gayan Gunarathne Technical Lead, WSO2 Inc. (http://wso2.com) Committer & PMC Member, Apache Stratos email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
