[ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhinandan Prateek updated CLOUDSTACK-4598: ------------------------------------------- Priority: Critical (was: Major) > [Performance Testing] High delays during deployVM - both network delay and > deployment planner delay > --------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-4598 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Environment: Simulator environment with large scale set up > Reporter: Sowmya Krishnan > Assignee: Murali Reddy > Priority: Critical > Fix For: 4.2.1 > > Attachments: deployVMjob_999.log.gz > > > This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these > issues were fixed and verified in comparatively smaller environment with 4K > and 8K hosts and 12K VMs > Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K > Pods. This is also a special case where we are trying to deploy one VM in > each host. > I am seeing delay both while acquiring network lock and during deployment > planning. > (There was also an ERROR observed in the log during deployment) > Log snippet: > 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing > clusters in order of aggregate capacity, that have (atleast one host with) > enough CPU and RAM capacity under this Zone: 1 > 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) > Removing from the clusterId list these clusters from avoid set: [] > .. > .. > 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) > Changing active number > of nics for network id=204 on 1 > 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking > VirtualRouter to prepare for > Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191] > 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed > to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] > 2013-09-02 22:51:04,702 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) > Cleaning up resources for the vm > VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] in Starting state > .. > .. > 2013-09-02 22:51:17,018 DEBUG [cloud.network.NetworkManagerImpl] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) > Changing active number of nics for network id=204 on 1 > 2013-09-02 22:51:17,074 DEBUG [cloud.network.NetworkManagerImpl] > (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking > VirtualRouter to prepare for > Nic[2246-1407-159bacce-8663-477e-ab37-2d1081c0630b-172.4.211.191] > 2013-09-02 22:57:56,139 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 > = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is acquired for network id > 204 as a part of router startup in > Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] > : > Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT-->Pool(9749))] > 2013-09-02 22:57:56,144 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 > = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is released for network id > 204 as a part of router startup in > Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] > : > Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT-->Pool(9749))] > .. > .. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira