I also get the following error when I try to launch an instance the tinylinux template. This time I was trying with NFS-based primary storage and uselocalstorage = false. I did not get this error earlier. The tinylinux template settings indicate that it is Centos 5.6 x86_64.
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-35100a3e) Add job-24 into job monitoring WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-14:ctx-e494228b) Task failed! Task record: uuid: a752017f-94a3-0682-28aa-62235059c010 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Thu Jan 16 07:42:55 PST 2014 finished: Thu Jan 16 07:42:55 PST 2014 status: failure residentOn: com.xensource.xenapi.Host@3ba61fa8 progress: 1.0 type: <none/> result: errorInfo: [VM_HVM_REQUIRED, OpaqueRef:93f3b6fa-27c1-4fe5-2185-3c724e087ae6] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] Thanks Prabhakaran On 1/16/14 1:20 PM, "Prabhakaran Ganesan" <gprab...@juniper.net> wrote: >All the while, I was using my own NFS based secondary storage and I Was >downloading the systemvm templates from jenkins. I made the secondary >storage point to 192.168.56.10:/opt/storage/secondary (I used the maven >deploySvr script to configure the whole thing). I edited the 'host' >parameter in GlobalSettings and restarted the management server. 'Host' >addition failed once, but the host got added after I rebooted the DevCloud >VM. The system VMs came up and I found that the tiny linux template got >downloaded. It also shows up when I try to launch an instance. > >When I launch the tiny linux instance, I run into the following error. The >router VM got started properly, but the linux instance didn't. It is >complaining about some storage issues. Any idea what is happening here? > >Thanks >Prabhakar > >INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-62fc8d1c) Add >job-14 into job monitoring >WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-12:ctx-45af0575) There >are 0 storageid: 9f3f9262-3f77-09cc-2df7-0d8475676260 >WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-56:ctx-cc6fb225) >Detecting a change in platform for r-8-VM >WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-18e096c3) Task (job-14) >has been pending for 98 seconds >WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-3:ctx-b1e5b734) There >are 0 storageid: 9f3f9262-3f77-09cc-2df7-0d8475676260 >INFO [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-1:ctx-62fc8d1c >ctx-29a1762b) lock is acquired for VMTemplateStoragePool 13 >WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-26:ctx-29532703) >There are 0 SRs with same name: 9f3f9262-3f77-09cc-2df7-0d8475676260 >INFO [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-1:ctx-62fc8d1c >ctx-29a1762b) releasing lock for VMTemplateStoragePool 13 >WARN [c.c.u.d.Merovingian2] (Job-Executor-1:ctx-62fc8d1c ctx-29a1762b) >Was unable to find lock for the key template_spool_ref13 and thread id >1518511247 >INFO [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-62fc8d1c >ctx-29a1762b) Unable to contact resource. >com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] >is unreachable: Unable to create Vol[7|vm=7|ROOT]:There are 0 SRs with >same name: 9f3f9262-3f77-09cc-2df7-0d8475676260 > at >org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolu >m >e(VolumeOrchestrator.java:1174) > at >org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(Volu >m >eOrchestrator.java:1216) > at >com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineMana >g >erImpl.java:944) > at >com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerI >m >pl.java:747) > at >com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.jav >a >:593) > at >org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVi >r >tualMachine(VMEntityManagerImpl.java:228) > at >org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.dep >l >oy(VirtualMachineEntityImpl.java:207) > at >com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java: >3 >572) > at >com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java: >3 >152) > at >com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java: >3 >138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: >5 >7) > at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm >p >l.java:43) > at java.lang.reflect.Method.invoke(Method.java:622) > at >org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Ao >p >Utils.java:317) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoi >n >t(ReflectiveMethodInvocation.java:183) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle >c >tiveMethodInvocation.java:150) > at >com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java: >5 >0) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle >c >tiveMethodInvocation.java:161) > at >org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Exp >o >seInvocationInterceptor.java:91) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle >c >tiveMethodInvocation.java:172) > at >org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopP >r >oxy.java:204) > at com.sun.proxy.$Proxy174.startVirtualMachine(Unknown Source) > at >org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd. >j >ava:443) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at >com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher. >j >ava:109) > at >com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De >f >aultManagedContext.java:56) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC >o >ntext(DefaultManagedContext.java:103) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo >n >text(DefaultManagedContext.java:53) > at >com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at >org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInConte >x >t(AsyncJobManagerImpl.java:524) > at >org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Managed >C >ontextRunnable.java:49) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De >f >aultManagedContext.java:56) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC >o >ntext(DefaultManagedContext.java:103) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo >n >text(DefaultManagedContext.java:53) > at >org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCo >n >textRunnable.java:46) > at >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: >1 >146) > at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java >: >615) > at java.lang.Thread.run(Thread.java:701) >INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-27:ctx-bf07d289) VM does >not exist on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea >WARN [o.a.c.alerts] (Job-Executor-1:ctx-62fc8d1c ctx-29a1762b) >alertType:: 8 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // >message:: Failed to deploy Vm with Id: 7, on Host with Id: null >ERROR [c.c.a.ApiAsyncJobDispatcher] (Job-Executor-1:ctx-62fc8d1c) >Unexpected exception while executing >org.apache.cloudstack.api.command.user.vm.DeployVMCmd >com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due >to insufficient capacity > at >com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.jav >a >:597) > at >org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVi >r >tualMachine(VMEntityManagerImpl.java:237) > at >org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.dep >l >oy(VirtualMachineEntityImpl.java:207) > at >com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java: >3 >572) > at >com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java: >3 >152) > at >com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java: >3 >138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: >5 >7) > at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm >p >l.java:43) > at java.lang.reflect.Method.invoke(Method.java:622) > at >org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Ao >p >Utils.java:317) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoi >n >t(ReflectiveMethodInvocation.java:183) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle >c >tiveMethodInvocation.java:150) > at >com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java: >5 >0) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle >c >tiveMethodInvocation.java:161) > at >org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Exp >o >seInvocationInterceptor.java:91) > at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle >c >tiveMethodInvocation.java:172) > at >org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopP >r >oxy.java:204) > at com.sun.proxy.$Proxy174.startVirtualMachine(Unknown Source) > at >org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd. >j >ava:443) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at >com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher. >j >ava:109) > at >com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De >f >aultManagedContext.java:56) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC >o >ntext(DefaultManagedContext.java:103) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo >n >text(DefaultManagedContext.java:53) > at >com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at >org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInConte >x >t(AsyncJobManagerImpl.java:524) > at >org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Managed >C >ontextRunnable.java:49) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De >f >aultManagedContext.java:56) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC >o >ntext(DefaultManagedContext.java:103) > at >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo >n >text(DefaultManagedContext.java:53) > at >org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCo >n >textRunnable.java:46) > at >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: >1 >146) > at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java >: >615) > at java.lang.Thread.run(Thread.java:701) >Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable >to create a deployment for >VM[User|VM-cf17832c-7957-4c67-a9bc-b4b04f6bbc69]Scope=interface >com.cloud.dc.DataCenter; id=1 > at >com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineMana >g >erImpl.java:903) > at >com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerI >m >pl.java:747) > at >com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.jav >a >:593) > ... 38 more >INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-62fc8d1c) Remove >job-14 from job monitoring > > > >On 1/16/14 12:28 PM, "Prabhakaran Ganesan" <gprab...@juniper.net> wrote: > >>Yes. The below script was executed on the SSVM and it was able to connect >>to the management server on port 8250. >> >><snip> >>Management server is 192.168.56.31. Checking connectivity. >>Good: Can connect to management server port 8250 >><\snip> >> >> >>Thanks >>Prabhakar >> >> >>On 1/16/14 12:24 PM, "Wei ZHOU" <ustcweiz...@gmail.com> wrote: >> >>>Can you ping 192.168.56.31 from SSVM ? >>> >>> >>>2014/1/16 Prabhakaran Ganesan <gprab...@juniper.net> >>> >>>> Yes, Edison. I restarted the management server. Here is the output of >>>>SSVM >>>> health script. >>>> >>>> root@s-2-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh >>>> ================================================ >>>> First DNS server is 192.168.56.1 >>>> PING 192.168.56.1 (192.168.56.1): 56 data bytes >>>> 64 bytes from 192.168.56.1: icmp_seq=0 ttl=64 time=0.842 ms >>>> 64 bytes from 192.168.56.1: icmp_seq=1 ttl=64 time=0.414 ms >>>> --- 192.168.56.1 ping statistics --- >>>> 2 packets transmitted, 2 packets received, 0% packet loss >>>> round-trip min/avg/max/stddev = 0.414/0.628/0.842/0.214 ms >>>> Good: Can ping DNS server >>>> ================================================ >>>> Good: DNS resolves download.cloud.com >>>> ================================================ >>>> nfs is currently mounted >>>> Mount point is /mnt/SecStorage/ece697f4-c6c2-323f-8148-e456cc6ed703 >>>> Good: Can write to mount point >>>> ================================================ >>>> Management server is 192.168.56.31. Checking connectivity. >>>> Good: Can connect to management server port 8250 >>>> ================================================ >>>> Good: Java process is running >>>> ================================================ >>>> Tests Complete. Look for ERROR or WARNING above. >>>> >>>> >>>> Thanks >>>> Prabakar >>>> >>>> On 1/16/14 11:56 AM, "Edison Su" <edison...@citrix.com> wrote: >>>> >>>> > >>>> >> -----Original Message----- >>>> >> From: Prabhakaran Ganesan [mailto:gprab...@juniper.net] >>>> >> Sent: Wednesday, January 15, 2014 9:34 PM >>>> >> To: dev@cloudstack.apache.org >>>> >> Subject: Re: Devcloud update for 4.3 >>>> >> >>>> >> Hi Edison >>>> >> >>>> >> Thanks for the clarification. I'll try to debug why the >>>>template >>>>is >>>> >>not >>>> >> downloaded. >>>> >> >>>> >> I'm running the management-server as a separate vm (over >>>>virtualBox, >>>> >> with one interface in the host-network) at 192.168.56.31. I have >>>>sent >>>> >>the >>>> >> 'host' parameter as 192.168.56.31. Isn't this correct? >>>> > >>>> >Did you restart mgt server, after set host equal to "192.168.56.31"? >>>> >If not, restart your mgt server, and stop/start ssvm >>>> >If yes, then the issue will be ssvm can't connect to >>>>192.168.56.31:8250, >>>> >do you open port 8250 on the VM where your mgt server running? >>>> > >>>> >> >>>> >> Thanks >>>> >> Prabhakar >>>> >> >>>> >> On 1/15/14 6:09 PM, "Edison Su" <edison...@citrix.com> wrote: >>>> >> >>>> >> > >>>> >> > >>>> >> >> -----Original Message----- >>>> >> >> From: Prabhakaran Ganesan [mailto:gprab...@juniper.net] >>>> >> >> Sent: Wednesday, January 15, 2014 4:28 PM >>>> >> >> To: dev@cloudstack.apache.org >>>> >> >> Subject: Re: Devcloud update for 4.3 >>>> >> >> >>>> >> >> Hi Edison >>>> >> >> >>>> >> >> Thanks for the details. I was trying to download the >>>>tty-linux >>>> >> >>iso and launch the same. Do you mean to use the built-in 'tiny >>>>linux >>>> >> >>template' by tty linux? >>>> >> >Yes, the build-in "tty linux". Doesn't it show up on the UI in >>>>create >>>> >> >vm wizard? >>>> >> >If not, then what's the agent status of your ssvm? If agent status >>>>is >>>> >> >not in up state(90% true), then check "host" in global >>>>configuration, >>>> >> >make sure "host = 192.168.56.10" >>>> >> > >>>> >> >> For some reason, it doesn't get downloaded in my environment. >>>> >> >> Is there anyway to download it to some location in sec storage >>>> >> >>(similar to downloading the system vm template)? >>>> >> >> >>>> >> >> Also, how do I launch the VM without 'data disk'. If I go >>>>through >>>> >> >>the UI, it always asks me to select a DiskOffering and I'm not >>>>able >>>> >> >>to create a 0GB custom offering also. >>>> >> >> >>>> >> >> Thanks >>>> >> >> Prabhakar >>>> >> >> >>>> >> >> On 1/15/14 4:15 PM, "Edison Su" <edison...@citrix.com> wrote: >>>> >> >> >>>> >> >> >Iso and data disk doesn't work on devcloud, as there is a bug >>>>in >>>> >> >> >debian xen kernel. The only thing devcloud can do, right now, >>>>is >>>> >> >> >launching tty Linux vm. >>>> >> >> >If you have a separate xen template, you can try to register >>>>the >>>> >> >> >template, then create a vm from it. But make sure, the template >>>>url >>>> >> >> >can be accessed by devcloud. >>>> >> >> > >>>> >> >> >> -----Original Message----- >>>> >> >> >> From: Prabhakaran Ganesan [mailto:gprab...@juniper.net] >>>> >> >> >> Sent: Wednesday, January 15, 2014 3:36 PM >>>> >> >> >> To: dev@cloudstack.apache.org >>>> >> >> >> Subject: Re: Devcloud update for 4.3 >>>> >> >> >> >>>> >> >> >> Hi Edison >>>> >> >> >> >>>> >> >> >> Thanks for your response. Yes, I'm trying to launch my >>>>own >>>> >> >> >>iso, with 'small' >>>> >> >> >> data disk offering. >>>> >> >> >> >>>> >> >> >> I tried registering the ttylinux VM ISO. What should I >>>>select >>>> >> >> >>for OS type? >>>> >> >> >> >>>> >> >> >> Thanks >>>> >> >> >> Prabhakaran >>>> >> >> >> >>>> >> >> >> On 1/15/14 3:23 PM, "Edison Su" <edison...@citrix.com> wrote: >>>> >> >> >> >>>> >> >> >> >Seems attach disk failed, are you using data disk offering >>>>when >>>> >> >> >> >you create the VM? How about just try to start ttyLinux vm, >>>> >> >> >> >without data >>>> >> >> >>disk. >>>> >> >> >> > >>>> >> >> >> >> -----Original Message----- >>>> >> >> >> >> From: Prabhakaran Ganesan [mailto:gprab...@juniper.net] >>>> >> >> >> >> Sent: Wednesday, January 15, 2014 3:10 PM >>>> >> >> >> >> To: dev@cloudstack.apache.org >>>> >> >> >> >> Subject: Re: Devcloud update for 4.3 >>>> >> >> >> >> >>>> >> >> >> >> I'm new to this alias, looks like it does not support >>>> >>attachments. >>>> >> >> >> >>I have pasted the significant error logs below. Please let >>>>me >>>> >> >> >> >>know how I can share the complete logs for your reference. >>>> >> >> >> >> >>>> >> >> >> >> Thanks >>>> >> >> >> >> Prabhakar >>>> >> >> >> >> >>>> >> >> >> >> Mgmt-server side: >>>> >> >> >> >> ================= >>>> >> >> >> >> [root@localhost cloudstack]# INFO >>>> >> >> >> >> [o.a.c.f.j.i.AsyncJobMonitor] >>>> >> >> >> >> (Job-Executor-11:ctx-686ff842) Add job-21 into job >>>>monitoring >>>> >> >> >> >> WARN [c.c.h.x.r.CitrixResourceBase] >>>> >>(DirectAgent-180:ctx-e20ff431) >>>> >> Task >>>> >> >> >> >> failed! Task record: uuid: >>>> >> >> >> >> 2640def4-6141-8698-ded9-adf2da19b746 >>>> >> >> >> >> nameLabel: Async.VM.start_on >>>> >> >> >> >> nameDescription: >>>> >> >> >> >> allowedOperations: [] >>>> >> >> >> >> currentOperations: {} >>>> >> >> >> >> created: Wed Jan 15 06:20:00 PST 2014 >>>> >> >> >> >> finished: Wed Jan 15 06:20:00 PST 2014 >>>> >> >> >> >> status: failure >>>> >> >> >> >> residentOn: com.xensource.xenapi.Host@3ba61fa8 >>>> >> >> >> >> progress: 1.0 >>>> >> >> >> >> type: <none/> >>>> >> >> >> >> result: >>>> >> >> >> >> errorInfo: [INTERNAL_ERROR, Xb.Noent] >>>> >> >> >> >> otherConfig: {} >>>> >> >> >> >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f >>>> >> >> >> >> subtasks: [] >>>> >> >> >> >> >>>> >> >> >> >> WARN [c.c.h.x.r.CitrixResourceBase] >>>> >> >> >> >>(DirectAgent-180:ctx-e20ff431) Unable to start >>>>VM(i-2-7-VM) >>>>on >>>> >> >> >> >>host(9d4c9db8-32f7-25c3-0435-eab4bf3adcea) due to >>>> >> >> >> >> Task failed! Task record: uuid: >>>> >> >> >> >> 2640def4-6141-8698-ded9-adf2da19b746 >>>> >> >> >> >> nameLabel: Async.VM.start_on >>>> >> >> >> >> nameDescription: >>>> >> >> >> >> allowedOperations: [] >>>> >> >> >> >> currentOperations: {} >>>> >> >> >> >> created: Wed Jan 15 06:20:00 PST 2014 >>>> >> >> >> >> finished: Wed Jan 15 06:20:00 PST 2014 >>>> >> >> >> >> status: failure >>>> >> >> >> >> residentOn: com.xensource.xenapi.Host@3ba61fa8 >>>> >> >> >> >> progress: 1.0 >>>> >> >> >> >> type: <none/> >>>> >> >> >> >> result: >>>> >> >> >> >> errorInfo: [INTERNAL_ERROR, Xb.Noent] >>>> >> >> >> >> otherConfig: {} >>>> >> >> >> >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f >>>> >> >> >> >> subtasks: [] >>>> >> >> >> >> >>>> >> >> >> >> Task failed! Task record: uuid: >>>> >> >> >> >> 2640def4-6141-8698-ded9-adf2da19b746 >>>> >> >> >> >> nameLabel: Async.VM.start_on >>>> >> >> >> >> nameDescription: >>>> >> >> >> >> allowedOperations: [] >>>> >> >> >> >> currentOperations: {} >>>> >> >> >> >> created: Wed Jan 15 06:20:00 PST 2014 >>>> >> >> >> >> finished: Wed Jan 15 06:20:00 PST 2014 >>>> >> >> >> >> status: failure >>>> >> >> >> >> residentOn: com.xensource.xenapi.Host@3ba61fa8 >>>> >> >> >> >> progress: 1.0 >>>> >> >> >> >> type: <none/> >>>> >> >> >> >> result: >>>> >> >> >> >> errorInfo: [INTERNAL_ERROR, Xb.Noent] >>>> >> >> >> >> otherConfig: {} >>>> >> >> >> >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f >>>> >> >> >> >> subtasks: [] >>>> >> >> >> >> >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSu >>>> >> c >>>> >> >> >> >>ce >>>> >> >> s >>>> >> >> >> >>s( >>>> >> >> >> C >>>> >> >> >> >>itr >>>> >> >> >> >> ix >>>> >> >> >> >> ResourceBase.java:3685) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(Cit >>>> >> >> >> >>rix >>>> >> >> >> >>Res >>>> >> >> >> >>ou >>>> >> >> >> >> rce >>>> >> >> >> >> Base.java:3797) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(Cit >>>> >> >> >> >>rix >>>> >> >> >> >>Res >>>> >> >> >> >>ou >>>> >> >> >> >> rce >>>> >> >> >> >> Base.java:1700) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeReq >>>> >> u >>>> >> >> >> >>e >>>> >> >> st >>>> >> >> >> >>( >>>> >> >> >> Ci >>>> >> >> >> >>tr >>>> >> >> >> >> ixR >>>> >> >> >> >> esourceBase.java:545) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >>>>com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOs >>>> >> >> >> >> sResou >>>> >> >> >> >> rce.java:133) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>com.cloud.agent.manager.DirectAgentAttache$Task.runInContext( >>>> >> Dir >>>> >> >> >> >>e >>>> >> >> ct >>>> >> >> >> Age >>>> >> >> >> >> ntAtt >>>> >> >> >> >> ache.java:216) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run( >>>> >> >> >> >> ManagedC >>>> >> >> >> >> ontextRunnable.java:49) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedCont >>>> >> ext >>>> >> >> $1. >>>> >> >> >> cal >>>> >> >> >> >> l(Def >>>> >> >> >> >> aultManagedContext.java:56) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedCont >>>> >> ext >>>> >> >> .ca >>>> >> >> >> >>l >>>> >> >> >> lW >>>> >> >> >> >> ithCo >>>> >> >> >> >> ntext(DefaultManagedContext.java:103) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run >>>> >> >> >> >> WithCon >>>> >> >> >> >> text(DefaultManagedContext.java:53) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Ma >>>> >> >> >> >> nagedCon >>>> >> >> >> >> textRunnable.java:46) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java: >>>> >> >> >>471 >>>> >> >> >>) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>> >> >> >> >> at >>>>java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutur >>>> >> e >>>> >> >> >> >>T >>>> >> >> ask. >>>> >> >> >> a >>>> >> >> >> >>c >>>> >> >> >> >> cess >>>> >> >> >> >> $101(ScheduledThreadPoolExecutor.java:165) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutur >>>> >> e >>>> >> >> >> >>T >>>> >> >> ask. >>>> >> >> >> r >>>> >> >> >> >>u >>>> >> >> >> >> n(Sc >>>> >> >> >> >> heduledThreadPoolExecutor.java:266) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExe >>>> >> c >>>> >> >> >> >>ut >>>> >> >> or. >>>> >> >> >> j >>>> >> >> >> >>av >>>> >> >> >> >> a:1 >>>> >> >> >> >> 146) >>>> >> >> >> >> at >>>> >> >> >> >> >>>> >> >> >> >>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx >>>> >> e >>>> >> >> >> >>c >>>> >> >> utor. >>>> >> >> >> >>ja >>>> >> >> >> >> va: >>>> >> >> >> >> 615) >>>> >> >> >> >> at java.lang.Thread.run(Thread.java:701) >>>> >> >> >> >> WARN [c.c.h.x.r.CitrixResourceBase] >>>> >> >> >> >>(DirectAgent-180:ctx-e20ff431) Catch >>>> >> >> >> >> Exception: class >>>> >> >> >> >>com.cloud.utils.exception.CloudRuntimeException >>>> >> >> >> >>due to >>>> >> >> >> >> com.cloud.utils.exception.CloudRuntimeException: Unable to >>>> >> >> >> >>start >>>> >> >> >> >> VM(i-2-7-VM) on host(9d4c9db8-32f7-25c3-0435-eab4bf3adcea) >>>> >> due >>>> >> >> to >>>> >> >> >> Task >>>> >> >> >> >> failed! Task record: uuid: >>>> >> >> >> >> 2640def4-6141-8698-ded9-adf2da19b746 >>>> >> >> >> >> nameLabel: Async.VM.start_on >>>> >> >> >> >> >>>> >> >> >> >> XCP-XAPI Errors: >>>> >> >> >> >> >>>> >> >> >> >> ================ >>>> >> >> >> >> 20140115T14:20:00.766Z| >>>>info|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|dispatcher] spawning a new thread to handle >>>>the >>>> >> >> >> >>current task (trackid=71ad5d3d9cf6b650d2105b1925aa94f7) >>>> >> >> >> >> >>>> >> [20140115T14:20:00.770Z|debug|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|audit] VM.start_on: VM = >>>> >> >> >> >> '85522f78-43c1-7668-cf2f-d5735262fce3 (i-2-7-VM)'; host >>>> >> >> >> >>'9d4c9db8-32f7- 25c3-0435-eab4bf3adcea (devcloud)' >>>> >> >> >> >> >>>> >> [20140115T14:20:00.780Z|debug|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|xapi] No operations are valid because >>>> >> >> >> >>current-operations = [ OpaqueRef:451bd4b9-39c8-2bcb-84b1- >>>> >> >> >> 03a0d7bd751d >>>> >> >> >> >>-> attach ] >>>> >> >> >> >>>>>>[20140115T14:20:00.789Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|xapi] host devcloud; available_memory = >>>> >> >> >> >>2757689344; memory_required = 575668224 >>>> >> >> >> >>>>>>[20140115T14:20:00.789Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|xapi_ha_vm_failover] >>>> >> >> >> >> assert_configuration_change_preserves_ha_plan c = >>>> >> >> >> >>configuration_change = { old_vms_leaving = [ ]; >>>> >> >> >> >>new_vms_arriving = [ 0d19f3fa (devcloud) >>>> >> >> >> >> 7eeed546 >>>> >> >> >> >> (i-2-7-VM) ]; hosts_to_disable = [ ]; num_failures = no >>>> >> >> >> >>change; new_vms = [ ] } >>>> >> >> >> >>>>>>[20140115T14:20:00.795Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|thread_queue] push(vm_lifecycle_op, >>>>VM.start >>>> >> >> >> >>Async.VM.start_on R:451bd4b939c8): queue = [ VM.start >>>> >> >> >> >>Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8 ](1) >>>> >> >> >> >> [20140115T14:20:00.795Z|debug|devcloud|186||thread_queue] >>>> >> >> >> >> pop(vm_lifecycle_op) = VM.start Async.VM.start_on >>>> >> >> >> >>R:451bd4b939c8 >>>> >> >> >> >>>>>>[20140115T14:20:00.796Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|locking_helpers] Acquired lock on VM >>>> >> >> >> >> OpaqueRef:7eeed546-7f37-ab5c-e495-772f4f52d7f8 with token >>>>14 >>>> >> >> >> >>>>>>[20140115T14:20:00.796Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|xapi] start: making sure the VM really is >>>>halted >>>> >> >> >> >>>>>>[20140115T14:20:00.796Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|xapi] start: checking that VM can run on >>>>this >>>> >> >> >> >>host [20140115T14:20:00.798Z| >>>> >> >> >> >>info|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|xapi] The VM's BIOS strings were not yet >>>>filled >>>> >>in. >>>> >> >> >> >>The VM is now made BIOS-generic. >>>> >> >> >> >> >>>> >> [20140115T14:20:00.799Z|debug|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|xapi] start: bringing up domain in the >>>>paused >>>> >> >> >> >>state >>>> >> >> >> >>>>>>[20140115T14:20:00.801Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|memory_control] reserve_memory_range >>>> >> >> min=562176 >>>> >> >> >> >> max=562176 >>>> >> >> >> >> >>>> >> [20140115T14:20:00.802Z|debug|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|xenstore-rpc] Checking pid 2734 >>>> >> >> >> >>>>>>[20140115T14:20:00.803Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|backtrace] Raised at hashtbl.ml:93.19-28 -> >>>> >> >> >> >> debug.ml:100.36-65 >>>> >> >> >> >> >>>>[20140115T14:20:00.804Z|error|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|memory_control] Unexpected exception >>>>talking >>>> >> to >>>> >> >> >> >>ballooning >>>> >> >> >> >> daemon: INTERNAL_ERROR: [ Xb.Noent ] >>>> >> >> >> >>>>>>[20140115T14:20:00.804Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|xapi] Raised at memory_control.ml:55.12-13 >>>>-> >>>> >> >> >> >> memory_control.ml:95.6-10 -> memory_control.ml:125.2-577 >>>>-> >>>> >> >> >> >> vmops.ml:994.4-261 -> pervasiveext.ml:22.2-9 >>>> >> >> >> >>>>>>[20140115T14:20:00.805Z|debug|devcloud|1461|Async.VM.start_o >>>> >> n >>>> >> >> >> >> R:451bd4b939c8|xapi] Raised at pervasiveext.ml:26.22-25 -> >>>> >> >> >> >> xenctrl.ml:114.13-17 -> xenctrl.ml:114.56-59 -> >>>> >> >> >> >>xapi_vm.ml:234.6-120 >>>> >> >> >> >>-> >>>> >> >> >> >> pervasiveext.ml:22.2-9 >>>> >> >> >> >> >>>> >> [20140115T14:20:00.805Z|debug|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|xapi] Raised at pervasiveext.ml:26.22-25 -> >>>> >> >> >> >> pervasiveext.ml:22.2-9 >>>> >> >> >> >> >>>> >> [20140115T14:20:00.806Z|debug|devcloud|1461|Async.VM.start_on >>>> >> >> >> >> R:451bd4b939c8|locking_helpers] Released lock on VM >>>> >> >> >> >> OpaqueRef:7eeed546-7f37-ab5c-e495-772f4f52d7f8 with token >>>>14 >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> On 1/15/14 3:03 PM, "Prabhakaran Ganesan" >>>> >> >> >> >> <gprab...@juniper.net> >>>> >> >> >> >> wrote: >>>> >> >> >> >> >>>> >> >> >> >> >Not sure if I attached the logs earlier. >>>> >> >> >> >> > >>>> >> >> >> >> >On 1/15/14 2:56 PM, "Prabhakaran Ganesan" >>>> >> >> >> >> ><gprab...@juniper.net> >>>> >> >> >> >> wrote: >>>> >> >> >> >> > >>>> >> >> >> >> >>Hi Edison >>>> >> >> >> >> >> >>>> >> >> >> >> >> I'm having trouble with launching guest instances. >>>>All >>>> >> >> >> >> >>system VMs are up and running, including the router. But >>>>when >>>> >> >> >> >> >>I try to launch the guest, I'm getting [INTERNAL_ERROR, >>>> >> Xb.noent]. >>>> >> >> >> >> >>Here is some more information on my environment. >>>> >> >> >> >> >> >>>> >> >> >> >> >>Host: VirtualBox 4.2 in Centos (192.168.56.1) runs NFS, >>>>DNS >>>> >> >> >> >> >>and WebServer Mgmt Server: Centos 6.4 VM >>>>(192.168.56.31), >>>> >> >> >> >> >>built from >>>> >> >> >> >> >>4.3 source (2G RAM, 20GB disk) >>>> >> >> >> >> >>DevCloud: Latest 4.3 compatible Devcloud2.ova downloaded >>>> >> from >>>> >> >> >> >> sebgoa's >>>> >> >> >> >> >>apache location (3.5G RAM, 30G disk) >>>> >> >> >> >> >> >>>> >> >> >> >> >>I have tried fixing Dom0 memory to allow more memory for >>>> >> >> >> >> >>guest VMs, as below. >>>> >> >> >> >> >> >>>> >> >> >> >> >>xe vm-param-set memory-dynamic-max=500MiB >>>> >> >> >> >> >>uuid=c8c2f1a3-40d3-308f-70bf-687f7f49b9a5 >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>Have attached the management-server and xcp-xapi logs >>>>here. I >>>> >> >> >> >> >>used to run into the same error with the Devcloud.ova >>>> >> >> >> >> >>downloaded from the bhaisaab's apache location. I have >>>>tried >>>> >> >> >> >> >>using different compute offerings (lower >>>> >> >> >> >> >>cpu/memory) also. Can you let me know what is going >>>>wrong >>>> >> here? >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>Thanks >>>> >> >> >> >> >>Prabhakar >>>> >> >> >> >> >> >>>> >> >> >> >> >>On 1/15/14 10:09 AM, "Edison Su" <edison...@citrix.com> >>>> >>wrote: >>>> >> >> >> >> >> >>>> >> >> >> >> >>>Thanks, could you update >>>> >> >> >> >> >>>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/De >>>> >> vCl >>>> >> >> oud >>>> >> >> >> >> >>>, >>>> >> >> >> >> >>>point to your link? >>>> >> >> >> >> >>>The devcloud reboot issue, also happened to me, haven't >>>>have >>>> >> >> >> >> >>>time to investigate it yet, it's only happened during >>>>the >>>> >> >> >> >> >>>first time adding the host though, and it can recover >>>> >> >> >> >> >>>itself, so not a >>>> >> >> >>critical bug. >>>> >> >> >> >> >>> >>>> >> >> >> >> >>>> -----Original Message----- >>>> >> >> >> >> >>>> From: sebgoa [mailto:run...@gmail.com] >>>> >> >> >> >> >>>> Sent: Wednesday, January 15, 2014 8:49 AM >>>> >> >> >> >> >>>> To: dev@cloudstack.apache.org >>>> >> >> >> >> >>>> Subject: Re: Devcloud update for 4.3 >>>> >> >> >> >> >>>> >>>> >> >> >> >> >>>> I put the image at >>>> >> >> >> >> >>>> http://people.apache.org/~sebgoa/devcloud2.ova >>>> >> >> >> >> >>>> >>>> >> >> >> >> >>>> >>>> >> >> >> >> >>>> >>>> >> >> >> >> >>>> On Jan 15, 2014, at 3:57 PM, sebgoa >>>><run...@gmail.com> >>>> >> wrote: >>>> >> >> >> >> >>>> >>>> >> >> >> >> >>>> > I confirm that this works. >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> > I am uploading the image to my people space on >>>>apache >>>> >> >>infra. >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> > I had several warnings, devcloud rebooted (which I >>>>don't >>>> >> >> >> >> >>>> > remember >>>> >> >> >> >> >>>> happening), but finally It seemed to recover and I >>>>started >>>> >> >> >> >> >>>>an instance. >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> > Here are the logs, if someone cares to look at >>>>them: >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> > http://pastebin.com/yR5RTJsj >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> > -sebastien >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> > On Jan 15, 2014, at 10:30 AM, Sebastien Goasguen >>>> >> >> >> >> >>>> > <run...@gmail.com> >>>> >> >> >> >> >>>> wrote: >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>>> >> >>>> >> >> >> >> >>>> >> On Jan 14, 2014, at 8:32 PM, Edison Su >>>> >> >> >> >> >>>> >> <edison...@citrix.com> >>>> >> >> >> >> >>>>wrote: >>>> >> >> >> >> >>>> >> >>>> >> >> >> >> >>>> >>> After several fixes, devcloud is working on 4.3 >>>>branch >>>> >> >>now. >>>> >> >> >> >> >>>> >>> Manually create a new devcloud image(the only >>>>change >>>> >> >> >> >> >>>> >>> to the image >>>> >> >> >> >> >>>>is >>>> >> >> >> >> >>>> to put 4.3 system vm template into >>>> >> >> >> >> >>>>/opt/storage/secondary/**/1/1/), and put it under: >>>> >> >> >> >> >>>>http://pan.baidu.com/s/1dDkmM5r (Can't find a free >>>>place >>>> to >>>> >> >> >> >> >>>>hold 1.3G data, so put it in a Chinese dropbox like >>>> >> >> >> >> >>>>website(which has >>>> >> >> >>2T >>>> >> >> >> storage for free...) ). >>>> >> >> >> >> >>>> >>> Not sure you can download it or not. If not, >>>>then >>>>I >>>> >> >> >> >> >>>> >>> may need help >>>> >> >> >> >> >>>>to find >>>> >> >> >> >> >>>> another place to store the image. >>>> >> >> >> >> >>>> >> >>>> >> >> >> >> >>>> >> Cool, >>>> >> >> >> >> >>>> >> >>>> >> >> >> >> >>>> >> Maybe you can put it on S3 or your public_html on >>>> >> >> >> >> >>>> >> people.apache.org >>>> >> >> >> >> >>>> >> >>>> >> >> >> >> >>>> >> >>>> >> >> >> >> >>>> > >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> > >>>> >> >> >> >> > >>>> >> >> >> >> > >>>> >> >> >> >> >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> >>>> >> > >>>> >> > >>>> >> > >>>> >> >>>> > >>>> > >>>> > >>>> >>>> >>>> >> >> >> >> > > > >