[jira] [Commented] (CLOUDSTACK-4598) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849018#comment-13849018 ] Murali Reddy commented on CLOUDSTACK-4598: -- Is this a regression in performance compared to 4.1? [Performance Testing] High delays during deployVM - both network delay and deployment planner delay --- Key: CLOUDSTACK-4598 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.2.0 Environment: Simulator environment with large scale set up Reporter: Sowmya Krishnan Assignee: Murali Reddy Priority: Critical Fix For: 4.3.0 Attachments: deployVMjob_999.log.gz This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these issues were fixed and verified in comparatively smaller environment with 4K and 8K hosts and 12K VMs Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K Pods. This is also a special case where we are trying to deploy one VM in each host. I am seeing delay both while acquiring network lock and during deployment planning. (There was also an ERROR observed in the log during deployment) Log snippet: 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Removing from the clusterId list these clusters from avoid set: [] .. .. 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191] 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] 2013-09-02 22:51:04,702 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Cleaning up resources for the vm VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] in Starting state .. .. 2013-09-02 22:51:17,018 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:51:17,074 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-159bacce-8663-477e-ab37-2d1081c0630b-172.4.211.191] 2013-09-02 22:57:56,139 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] 2013-09-02 22:57:56,144 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] .. .. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (CLOUDSTACK-4598) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790234#comment-13790234 ] Abhinandan Prateek commented on CLOUDSTACK-4598: Performance issue moving to 4.3 [Performance Testing] High delays during deployVM - both network delay and deployment planner delay --- Key: CLOUDSTACK-4598 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.2.0 Environment: Simulator environment with large scale set up Reporter: Sowmya Krishnan Assignee: Murali Reddy Priority: Critical Fix For: 4.3.0 Attachments: deployVMjob_999.log.gz This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these issues were fixed and verified in comparatively smaller environment with 4K and 8K hosts and 12K VMs Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K Pods. This is also a special case where we are trying to deploy one VM in each host. I am seeing delay both while acquiring network lock and during deployment planning. (There was also an ERROR observed in the log during deployment) Log snippet: 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Removing from the clusterId list these clusters from avoid set: [] .. .. 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191] 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] 2013-09-02 22:51:04,702 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Cleaning up resources for the vm VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] in Starting state .. .. 2013-09-02 22:51:17,018 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:51:17,074 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-159bacce-8663-477e-ab37-2d1081c0630b-172.4.211.191] 2013-09-02 22:57:56,139 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] 2013-09-02 22:57:56,144 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] .. .. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CLOUDSTACK-4598) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757588#comment-13757588 ] Koushik Das commented on CLOUDSTACK-4598: - 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191] 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] There must be some exceptions due to which the deployment failed. Can you provide those as well? [Performance Testing] High delays during deployVM - both network delay and deployment planner delay --- Key: CLOUDSTACK-4598 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.2.0 Environment: Simulator environment with large scale set up Reporter: Sowmya Krishnan Fix For: 4.2.1 Attachments: deployVMjob_999.log.gz This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these issues were fixed and verified in comparatively smaller environment with 4K and 8K hosts and 12K VMs Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K Pods. This is also a special case where we are trying to deploy one VM in each host. I am seeing delay both while acquiring network lock and during deployment planning. (There was also an ERROR observed in the log during deployment) Log snippet: 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Removing from the clusterId list these clusters from avoid set: [] .. .. 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191] 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] 2013-09-02 22:51:04,702 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Cleaning up resources for the vm VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] in Starting state .. .. 2013-09-02 22:51:17,018 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:51:17,074 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-159bacce-8663-477e-ab37-2d1081c0630b-172.4.211.191] 2013-09-02 22:57:56,139 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] 2013-09-02 22:57:56,144 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] .. .. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CLOUDSTACK-4598) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757612#comment-13757612 ] Sowmya Krishnan commented on CLOUDSTACK-4598: - Sorry - that must've got ripped off during grep. Error is concerning the Network Lock: 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] com.cloud.exception.ConcurrentOperationException: Unable to lock network 204 at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.findOrDeployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1432) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1947) at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:221) at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2057) at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2191) at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127) at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay --- Key: CLOUDSTACK-4598 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.2.0 Environment: Simulator environment with large scale set up Reporter: Sowmya Krishnan Fix For: 4.2.1 Attachments: deployVMjob_999.log.gz This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these issues were fixed and verified in comparatively smaller environment with 4K and 8K hosts and 12K VMs Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K Pods. This is also a special case where we are trying to deploy one VM in each host. I am seeing delay both while acquiring network lock and during deployment planning. (There was also an ERROR observed in the log during deployment) Log snippet: 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Removing from the clusterId list these clusters from avoid set: [] .. .. 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1
[jira] [Commented] (CLOUDSTACK-4598) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757650#comment-13757650 ] Sowmya Krishnan commented on CLOUDSTACK-4598: - Just to add the VM deployment finally succeeded... (for job-999) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay --- Key: CLOUDSTACK-4598 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.2.0 Environment: Simulator environment with large scale set up Reporter: Sowmya Krishnan Assignee: Murali Reddy Priority: Critical Fix For: 4.2.1 Attachments: deployVMjob_999.log.gz This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these issues were fixed and verified in comparatively smaller environment with 4K and 8K hosts and 12K VMs Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K Pods. This is also a special case where we are trying to deploy one VM in each host. I am seeing delay both while acquiring network lock and during deployment planning. (There was also an ERROR observed in the log during deployment) Log snippet: 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Removing from the clusterId list these clusters from avoid set: [] .. .. 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191] 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] 2013-09-02 22:51:04,702 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Cleaning up resources for the vm VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f] in Starting state .. .. 2013-09-02 22:51:17,018 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204 on 1 2013-09-02 22:51:17,074 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-159bacce-8663-477e-ab37-2d1081c0630b-172.4.211.191] 2013-09-02 22:57:56,139 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] 2013-09-02 22:57:56,144 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999 = [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))] : Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT--Pool(9749))] .. .. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira