[
https://issues.apache.org/jira/browse/CLOUDSTACK-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735615#comment-13735615
]
Kelven Yang commented on CLOUDSTACK-3568:
-----------------------------------------
We need to manage a pool of VmwareContext objects and manage API calls to
vCenter through acquired VmwareContext object to eliminate sharing of vCenter
session among multiple threads.
I'll work on this change and hopefully to make it available as soon as possible
> [Automation]Parallel deployment - Vmware - When deploying 30 parallel Vms ,
> we see ~15 Vms fail in org.apache.cloudstack.storage.command.CopyCommand.
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3568
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3568
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server, VMware
> Affects Versions: 4.2.0
> Environment: Build from 4.2
> Reporter: Sangeetha Hariharan
> Assignee: Kelven Yang
> Priority: Blocker
> Fix For: 4.2.0
>
> Attachments: management-server.log
>
>
> Parallel deployment - Vmware - When deploying 30 parallel Vms , we see ~15
> Vms fail in org.apache.cloudstack.storage.command.CopyCommand.
> Steps to reproduce the problem:
> Set up - Advanced zone with Vmware ESXI, 5.0.0 host.
> Deploy 30 Vms in parallel in a network that is already implemented (router is
> already running).
> Only 9 vms were successfully deployed.
> ~15 Vms failed in org.apache.cloudstack.storage.command.CopyCommand because
> of "ROOT-106-104/ROOT-106-104.vmdk was not found" error.
> mysql> select count(*),state,type from vm_instance where name like "hello%"
> group by state,type ;
> +----------+---------+------+
> | count(*) | state | type |
> +----------+---------+------+
> | 42 | Error | User |
> | 9 | Running | User |
> +----------+---------+------+
> 2 rows in set (0.00 sec)
> Following exception seen in management server logs:
> 2013-07-16 15:47:55,099 ERROR [storage.resource.VmwareStorageProcessor]
> (DirectAgent-71:10.223.57.66) CreateCommand failed due to Exception:
> java.lang.RuntimeException
> Message: File [7caed722555d3f71879476c5e8d8d5ad]
> ROOT-106-104/ROOT-106-104.vmdk was not found
> java.lang.RuntimeException: File [7caed722555d3f71879476c5e8d8d5ad]
> ROOT-106-104/ROOT-106-104.vmdk was not found
> at
> com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
> at
> com.cloud.hypervisor.vmware.mo.HostDatastoreBrowserMO.searchDatastore(HostDatastoreBrowserMO.java:57)
> at
> com.cloud.hypervisor.vmware.mo.HostDatastoreBrowserMO.searchDatastore(HostDatastoreBrowserMO.java:78)
> at
> com.cloud.hypervisor.vmware.mo.DatastoreMO.folderExists(DatastoreMO.java:319)
> at
> com.cloud.storage.resource.VmwareStorageProcessor.createVMLinkedClone(VmwareStorageProcessor.java:279)
> at
> com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:393)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:73)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:565)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-16 15:47:55,100 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-71:null) Seq 1-227737729: Response Received:
> 2013-07-16 15:47:55,100 DEBUG [agent.transport.Request] (DirectAgent-71:null)
> Seq 1-227737729: Processing: { Ans: , MgmtId: 7647994577963, via: 1, Ver:
> v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException:
> File [7caed722555d3f71879476c5e8d8d5ad] ROOT-106-104/ROOT-106-104.vmdk was
> not found","wait":0}}] }
> 2013-07-16 15:47:55,100 DEBUG [agent.transport.Request]
> (Job-Executor-8:job-101 = [ cad3c4ea-3d40-46f4-bce5-26d60d4af7c1 ]) Seq
> 1-227737729: Received: { Ans: , MgmtId: 7647994577963, via: 1, Ver: v1,
> Flags: 10, { CopyCmdAnswer } }
> mysql> select count(*),state,type from vm_instance where name like "hello%"
> group by state,type ;
> +----------+---------+------+
> | count(*) | state | type |
> +----------+---------+------+
> | 42 | Error | User |
> | 9 | Running | User |
> +----------+---------+------+
> 2 rows in set (0.00 sec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
