Looks like I'll have to dig up that doc on debugging SSVM...it's been about 30 mins since the VMs came up.
On Tue, Oct 8, 2013 at 7:29 PM, Marcus Sorensen <shadow...@gmail.com> wrote: > You may just have to wait a bit. That means the VMS are started but the > services aren't up. Usually takes 5-10 min to get both up in my fusion vm. > If that doesn't get better you may have to log into the system VMS and see > what's going on. > On Oct 8, 2013 7:25 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> > wrote: > >> Doh! I didn't restart the necessary service after exporting my NFS share. >> >> The system VMs come up now; however, SSVM has a dash (-) for its Agent >> State and the CS MS console keeps printing out the following: >> >> INFO [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-3:ctx-018adc41) >> No running ssvm is found, so command will be sent to LocalHostEndPoint >> >> >> On Tue, Oct 8, 2013 at 4:35 PM, Marcus Sorensen <shadow...@gmail.com>wrote: >> >>> Can you mount the secondary storage from your KVM host? >>> On Oct 8, 2013 4:01 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> >>> wrote: >>> >>>> Although the host is added to KVM, I do see the following issues in the >>>> CS MS console (any thoughts on this?): >>>> >>>> WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-c1c573ee) Was unable to >>>> find lock for the key template_spool_ref2 and thread id 2049868806 >>>> INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-c1c573ee) >>>> Unable to contact resource. >>>> com.cloud.exception.StorageUnavailableException: Resource >>>> [StoragePool:1] is unreachable: Unable to create >>>> Vol[1|vm=1|ROOT]:com.cloud.utils.exception.CloudRuntimeException: >>>> org.libvirt.LibvirtException: internal error Child process (/bin/mount >>>> 192.168.233.10:/mnt/secondary/template/tmpl/1/3 >>>> /mnt/334b3c4e-764b-362a-be2c-ebe8c490d0a9) status unexpected: exit status >>>> 32 >>>> at >>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1027) >>>> at >>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1069) >>>> at >>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:830) >>>> at >>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:649) >>>> at >>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261) >>>> at >>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:693) >>>> at >>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265) >>>> at >>>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123) >>>> at >>>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) >>>> at >>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101) >>>> at >>>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) >>>> at >>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78) >>>> at >>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:724) >>>> >>>> >>>> On Tue, Oct 8, 2013 at 3:58 PM, Mike Tutkowski < >>>> mike.tutkow...@solidfire.com> wrote: >>>> >>>>> So...got some good news: >>>>> >>>>> I spent a couple hours setting up a KVM environment on Ubuntu 12.04.1 >>>>> from scratch (Installing SSH, Open iSCSI, Java 7, KVM, Git, CloudStack, >>>>> CloudStack DEBs, KVM system template, etc.) and I can now add this KVM >>>>> host >>>>> to CloudStack (on a related note, no errors in agent.err either). >>>>> >>>>> I have no idea what is messed up with my old KVM install on Ubuntu, >>>>> but the new one works. >>>>> >>>>> That being the case, I can close out the JIRA ticket I logged a while >>>>> back and start integrating your code into mine. >>>>> >>>>> >>>>> On Mon, Oct 7, 2013 at 7:46 PM, Mike Tutkowski < >>>>> mike.tutkow...@solidfire.com> wrote: >>>>> >>>>>> Right...yeah, I didn't mean we'd commit to 4.2, but maybe I should >>>>>> work off of 4.2 since master seems to be un-stable in this regard. >>>>>> >>>>>> I plan to set up a machine in the lab tomorrow with Ubuntu 12.04 from >>>>>> scratch to see if it works when I start clean, but - if it doesn't - I >>>>>> should just use 4.2 for development. >>>>>> >>>>>> >>>>>> On Mon, Oct 7, 2013 at 7:05 PM, Marcus Sorensen >>>>>> <shadow...@gmail.com>wrote: >>>>>> >>>>>>> We can't. This patch will never see 4.2. You can still start working >>>>>>> on your plugin on 4.2, but the change represented by this patch can >>>>>>> only go >>>>>>> into master. >>>>>>> On Oct 7, 2013 5:01 PM, "Mike Tutkowski" < >>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>> >>>>>>>> So, now that I'm getting back to this, do you think I should just >>>>>>>> try to make this work with 4.2 (like we originally talked about)? >>>>>>>> >>>>>>>> I updated again from master, rebuilt, redeployed DEBs and still get >>>>>>>> this JNA error message: >>>>>>>> >>>>>>>> log4j:WARN No appenders could be found for logger >>>>>>>> (org.apache.commons.httpclient.params.DefaultHttpParams). >>>>>>>> log4j:WARN Please initialize the log4j system properly. >>>>>>>> log4j:WARN See >>>>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more >>>>>>>> info. >>>>>>>> java.lang.reflect.InvocationTargetException >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> at >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>> at >>>>>>>> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) >>>>>>>> Caused by: java.lang.UnsatisfiedLinkError: Can't obtain >>>>>>>> updateLastError method for class com.sun.jna.Native >>>>>>>> at com.sun.jna.Native.initIDs(Native Method) >>>>>>>> at com.sun.jna.Native.<clinit>(Native.java:139) >>>>>>>> at org.libvirt.jna.Libvirt.<clinit>(Unknown Source) >>>>>>>> at org.libvirt.Library.<clinit>(Unknown Source) >>>>>>>> at org.libvirt.Connect.<init>(Unknown Source) >>>>>>>> at >>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:44) >>>>>>>> at >>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:37) >>>>>>>> at >>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:733) >>>>>>>> at com.cloud.agent.Agent.<init>(Agent.java:161) >>>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415) >>>>>>>> at >>>>>>>> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:370) >>>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) >>>>>>>> at com.cloud.agent.AgentShell.start(AgentShell.java:448) >>>>>>>> ... 5 more >>>>>>>> Cannot start daemon >>>>>>>> Service exit with a return value of 5 >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 7, 2013 at 2:31 PM, Mike Tutkowski < >>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>> >>>>>>>>> Sure, that's a good plan. >>>>>>>>> >>>>>>>>> I'll get to it. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Oct 7, 2013 at 2:29 PM, Marcus Sorensen < >>>>>>>>> shadow...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I know you mentioned you might need some minor changes to it, as >>>>>>>>>> well as other minor changes just for master (attach volume switched >>>>>>>>>> to pool >>>>>>>>>> vs adapter or something). My hope was that you would be able to send >>>>>>>>>> an >>>>>>>>>> update that works for your plugin on master, I'll test against >>>>>>>>>> existing >>>>>>>>>> libvirtd storage and apply it. >>>>>>>>>> On Oct 7, 2013 1:49 PM, "Mike Tutkowski" < >>>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>>> >>>>>>>>>>> This is an automatically generated e-mail. To reply, visit: >>>>>>>>>>> https://reviews.apache.org/r/14381/ >>>>>>>>>>> >>>>>>>>>>> This looks reasonable to me, Marcus. >>>>>>>>>>> >>>>>>>>>>> When do you think you might start the process of getting this into >>>>>>>>>>> master? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - Mike Tutkowski >>>>>>>>>>> >>>>>>>>>>> On September 30th, 2013, 5:14 p.m. UTC, Marcus Sorensen wrote: >>>>>>>>>>> Review request for cloudstack, edison su and Mike Tutkowski. >>>>>>>>>>> By Marcus Sorensen. >>>>>>>>>>> >>>>>>>>>>> *Updated Sept. 30, 2013, 5:14 p.m.* >>>>>>>>>>> *Repository: * cloudstack-git >>>>>>>>>>> Description >>>>>>>>>>> >>>>>>>>>>> With custom storage plugins comes the need to prep the KVM host >>>>>>>>>>> prior to utilizing the disks. e.g. an iscsi initiator needs to log >>>>>>>>>>> into the target and scan for the lun before it can be used on the >>>>>>>>>>> host. This patch is an example I developed against 4.2, minor >>>>>>>>>>> changes may be necessary to apply to master, but I want to share >>>>>>>>>>> with others who are working on storage so they can ensure it works >>>>>>>>>>> for them. Please tweak as you see fit. >>>>>>>>>>> >>>>>>>>>>> MigrateCommand: pass vmTO object so we can see which disks/storage >>>>>>>>>>> pool types belong to the vm when migrating a VM. This facilitates >>>>>>>>>>> being able to call disconnectPhysicalDisksViaVmSpec >>>>>>>>>>> >>>>>>>>>>> VirtualMachineManagerImpl: pass VirtualMachineTO when migrating so >>>>>>>>>>> that we can see which disks belong to the VM and what storage >>>>>>>>>>> pools/adaptors should be used >>>>>>>>>>> >>>>>>>>>>> LibvirtComputingResource: add calls KVMStoragePoolManager's >>>>>>>>>>> connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec >>>>>>>>>>> calls where appropriate (when starting a vm, migrating a vm). >>>>>>>>>>> Ensure that we create 'raw' format XML disk definitions when the >>>>>>>>>>> storage format is RAW. Move cleanupDisk logic to storage adaptors >>>>>>>>>>> so that each adaptor type can clean up its disks in is own way. >>>>>>>>>>> >>>>>>>>>>> KVMStoragePoolManager: add connectPhysicalDisk, >>>>>>>>>>> disconnectPhysicalDisk, connectPhysicalDiskViaVmSpec, >>>>>>>>>>> disconnectPhysicalDiskViaVmSpec, disconnectPhysicalDiskByPath. >>>>>>>>>>> These all call the specific StorageAdaptor's connectPhysicalDisk, >>>>>>>>>>> disconnectPhysicalDisk, or disconnectPhysicalDiskByPath calls. >>>>>>>>>>> >>>>>>>>>>> KVMStorageProcessor: Call >>>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk on the storage adaptor. >>>>>>>>>>> Whether or not this is implemented is up to the storage adaptor. >>>>>>>>>>> >>>>>>>>>>> LibvirtStorageAdaptor: implement dummy >>>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk, move cleanupDisk logic >>>>>>>>>>> from LibvirtComputingResource to disconnectPhysicalDiskByPath >>>>>>>>>>> >>>>>>>>>>> StorageAdaptor: define >>>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk/disconnectPhysicalDiskByPath >>>>>>>>>>> in the interface >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Testing >>>>>>>>>>> >>>>>>>>>>> Basic testing with my storage adaptor >>>>>>>>>>> >>>>>>>>>>> Diffs >>>>>>>>>>> >>>>>>>>>>> - core/src/com/cloud/agent/api/MigrateCommand.java (5042b8c) >>>>>>>>>>> - >>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java >>>>>>>>>>> (3ee811f) >>>>>>>>>>> - >>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java >>>>>>>>>>> (e09c9ba) >>>>>>>>>>> - >>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java >>>>>>>>>>> (c69f9b0) >>>>>>>>>>> - >>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java >>>>>>>>>>> (123a9f1) >>>>>>>>>>> - >>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java >>>>>>>>>>> (4956d8d) >>>>>>>>>>> - server/src/com/cloud/vm/VirtualMachineManagerImpl.java >>>>>>>>>>> (d46bbb0) >>>>>>>>>>> >>>>>>>>>>> View Diff <https://reviews.apache.org/r/14381/diff/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Mike Tutkowski* >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>> o: 303.746.7302 >>>>>>>>> Advancing the way the world uses the >>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>>> *™* >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Mike Tutkowski* >>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>> o: 303.746.7302 >>>>>>>> Advancing the way the world uses the >>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>> *™* >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Mike Tutkowski* >>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> e: mike.tutkow...@solidfire.com >>>>>> o: 303.746.7302 >>>>>> Advancing the way the world uses the >>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>> *™* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the >>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>> *™* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the >>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>> *™* >>>> >>> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the >> cloud<http://solidfire.com/solution/overview/?video=play> >> *™* >> > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
