Might be this one: * https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting
Travis On Oct 8, 2013, at 11:04 PM, Mike Tutkowski <mike.tutkow...@solidfire.com> wrote: > CS MS can ping the SSVM's public IP address. > > SSVM can ping CS MS. > > I think we had a nice debug document for SSVM mentioned on a CS > e-mail...I'll try to find it. > > > On Tue, Oct 8, 2013 at 8:29 PM, Marcus Sorensen <shadow...@gmail.com> wrote: > >> You could start by trying to ping them from mgmt server, or trying to VNC >> to them if the OS isn't up. I'm assuming you prepped a current system vm >> template (maybe you're even reusing a secondary storage). If you can ping >> their 169.254 addresses, you can "cloudstack-ssh 169.254.x.x" as root on >> the kvm host to get in >> On Oct 8, 2013 7:31 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> >> wrote: >> >>> Looks like I'll have to dig up that doc on debugging SSVM...it's been >>> about 30 mins since the VMs came up. >>> >>> >>> On Tue, Oct 8, 2013 at 7:29 PM, Marcus Sorensen <shadow...@gmail.com>wrote: >>> >>>> You may just have to wait a bit. That means the VMS are started but the >>>> services aren't up. Usually takes 5-10 min to get both up in my fusion vm. >>>> If that doesn't get better you may have to log into the system VMS and see >>>> what's going on. >>>> On Oct 8, 2013 7:25 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> >>>> wrote: >>>> >>>>> Doh! I didn't restart the necessary service after exporting my NFS >>>>> share. >>>>> >>>>> The system VMs come up now; however, SSVM has a dash (-) for its Agent >>>>> State and the CS MS console keeps printing out the following: >>>>> >>>>> INFO [o.a.c.s.e.DefaultEndPointSelector] >>>>> (StatsCollector-3:ctx-018adc41) No running ssvm is found, so command will >>>>> be sent to LocalHostEndPoint >>>>> >>>>> >>>>> On Tue, Oct 8, 2013 at 4:35 PM, Marcus Sorensen >>>>> <shadow...@gmail.com>wrote: >>>>> >>>>>> Can you mount the secondary storage from your KVM host? >>>>>> On Oct 8, 2013 4:01 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> >>>>>> wrote: >>>>>> >>>>>>> Although the host is added to KVM, I do see the following issues in >>>>>>> the CS MS console (any thoughts on this?): >>>>>>> >>>>>>> WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-c1c573ee) Was unable >>>>>>> to find lock for the key template_spool_ref2 and thread id 2049868806 >>>>>>> INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-c1c573ee) >>>>>>> Unable to contact resource. >>>>>>> com.cloud.exception.StorageUnavailableException: Resource >>>>>>> [StoragePool:1] is unreachable: Unable to create >>>>>>> Vol[1|vm=1|ROOT]:com.cloud.utils.exception.CloudRuntimeException: >>>>>>> org.libvirt.LibvirtException: internal error Child process (/bin/mount >>>>>>> 192.168.233.10:/mnt/secondary/template/tmpl/1/3 >>>>>>> /mnt/334b3c4e-764b-362a-be2c-ebe8c490d0a9) status unexpected: exit >>>>>>> status 32 >>>>>>> at >>>>>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1027) >>>>>>> at >>>>>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1069) >>>>>>> at >>>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:830) >>>>>>> at >>>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:649) >>>>>>> at >>>>>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261) >>>>>>> at >>>>>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:693) >>>>>>> at >>>>>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265) >>>>>>> at >>>>>>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123) >>>>>>> at >>>>>>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) >>>>>>> at >>>>>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101) >>>>>>> at >>>>>>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) >>>>>>> at >>>>>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78) >>>>>>> at >>>>>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71) >>>>>>> at >>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>>> at >>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>>> at >>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:724) >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 8, 2013 at 3:58 PM, Mike Tutkowski < >>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>> >>>>>>>> So...got some good news: >>>>>>>> >>>>>>>> I spent a couple hours setting up a KVM environment on Ubuntu >>>>>>>> 12.04.1 from scratch (Installing SSH, Open iSCSI, Java 7, KVM, Git, >>>>>>>> CloudStack, CloudStack DEBs, KVM system template, etc.) and I can now >>>>>>>> add >>>>>>>> this KVM host to CloudStack (on a related note, no errors in agent.err >>>>>>>> either). >>>>>>>> >>>>>>>> I have no idea what is messed up with my old KVM install on Ubuntu, >>>>>>>> but the new one works. >>>>>>>> >>>>>>>> That being the case, I can close out the JIRA ticket I logged a >>>>>>>> while back and start integrating your code into mine. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 7, 2013 at 7:46 PM, Mike Tutkowski < >>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>> >>>>>>>>> Right...yeah, I didn't mean we'd commit to 4.2, but maybe I should >>>>>>>>> work off of 4.2 since master seems to be un-stable in this regard. >>>>>>>>> >>>>>>>>> I plan to set up a machine in the lab tomorrow with Ubuntu 12.04 >>>>>>>>> from scratch to see if it works when I start clean, but - if it >>>>>>>>> doesn't - I >>>>>>>>> should just use 4.2 for development. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Oct 7, 2013 at 7:05 PM, Marcus Sorensen < >>>>>>>>> shadow...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> We can't. This patch will never see 4.2. You can still start >>>>>>>>>> working on your plugin on 4.2, but the change represented by this >>>>>>>>>> patch can >>>>>>>>>> only go into master. >>>>>>>>>> On Oct 7, 2013 5:01 PM, "Mike Tutkowski" < >>>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>>> >>>>>>>>>>> So, now that I'm getting back to this, do you think I should just >>>>>>>>>>> try to make this work with 4.2 (like we originally talked about)? >>>>>>>>>>> >>>>>>>>>>> I updated again from master, rebuilt, redeployed DEBs and still >>>>>>>>>>> get this JNA error message: >>>>>>>>>>> >>>>>>>>>>> log4j:WARN No appenders could be found for logger >>>>>>>>>>> (org.apache.commons.httpclient.params.DefaultHttpParams). >>>>>>>>>>> log4j:WARN Please initialize the log4j system properly. >>>>>>>>>>> log4j:WARN See >>>>>>>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more >>>>>>>>>>> info. >>>>>>>>>>> java.lang.reflect.InvocationTargetException >>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>> at >>>>>>>>>>> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) >>>>>>>>>>> Caused by: java.lang.UnsatisfiedLinkError: Can't obtain >>>>>>>>>>> updateLastError method for class com.sun.jna.Native >>>>>>>>>>> at com.sun.jna.Native.initIDs(Native Method) >>>>>>>>>>> at com.sun.jna.Native.<clinit>(Native.java:139) >>>>>>>>>>> at org.libvirt.jna.Libvirt.<clinit>(Unknown Source) >>>>>>>>>>> at org.libvirt.Library.<clinit>(Unknown Source) >>>>>>>>>>> at org.libvirt.Connect.<init>(Unknown Source) >>>>>>>>>>> at >>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:44) >>>>>>>>>>> at >>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:37) >>>>>>>>>>> at >>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:733) >>>>>>>>>>> at com.cloud.agent.Agent.<init>(Agent.java:161) >>>>>>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415) >>>>>>>>>>> at >>>>>>>>>>> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:370) >>>>>>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) >>>>>>>>>>> at com.cloud.agent.AgentShell.start(AgentShell.java:448) >>>>>>>>>>> ... 5 more >>>>>>>>>>> Cannot start daemon >>>>>>>>>>> Service exit with a return value of 5 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 7, 2013 at 2:31 PM, Mike Tutkowski < >>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Sure, that's a good plan. >>>>>>>>>>>> >>>>>>>>>>>> I'll get to it. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Oct 7, 2013 at 2:29 PM, Marcus Sorensen < >>>>>>>>>>>> shadow...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I know you mentioned you might need some minor changes to it, >>>>>>>>>>>>> as well as other minor changes just for master (attach volume >>>>>>>>>>>>> switched to >>>>>>>>>>>>> pool vs adapter or something). My hope was that you would be able >>>>>>>>>>>>> to send >>>>>>>>>>>>> an update that works for your plugin on master, I'll test against >>>>>>>>>>>>> existing >>>>>>>>>>>>> libvirtd storage and apply it. >>>>>>>>>>>>> On Oct 7, 2013 1:49 PM, "Mike Tutkowski" < >>>>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> This is an automatically generated e-mail. To reply, >>>>>>>>>>>>>> visit: https://reviews.apache.org/r/14381/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> This looks reasonable to me, Marcus. >>>>>>>>>>>>>> >>>>>>>>>>>>>> When do you think you might start the process of getting this >>>>>>>>>>>>>> into master? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Mike Tutkowski >>>>>>>>>>>>>> >>>>>>>>>>>>>> On September 30th, 2013, 5:14 p.m. UTC, Marcus Sorensen wrote: >>>>>>>>>>>>>> Review request for cloudstack, edison su and Mike Tutkowski. >>>>>>>>>>>>>> By Marcus Sorensen. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Updated Sept. 30, 2013, 5:14 p.m.* >>>>>>>>>>>>>> *Repository: * cloudstack-git >>>>>>>>>>>>>> Description >>>>>>>>>>>>>> >>>>>>>>>>>>>> With custom storage plugins comes the need to prep the KVM host >>>>>>>>>>>>>> prior to utilizing the disks. e.g. an iscsi initiator needs to >>>>>>>>>>>>>> log into the target and scan for the lun before it can be used >>>>>>>>>>>>>> on the host. This patch is an example I developed against 4.2, >>>>>>>>>>>>>> minor changes may be necessary to apply to master, but I want to >>>>>>>>>>>>>> share with others who are working on storage so they can ensure >>>>>>>>>>>>>> it works for them. Please tweak as you see fit. >>>>>>>>>>>>>> >>>>>>>>>>>>>> MigrateCommand: pass vmTO object so we can see which >>>>>>>>>>>>>> disks/storage pool types belong to the vm when migrating a VM. >>>>>>>>>>>>>> This facilitates being able to call >>>>>>>>>>>>>> disconnectPhysicalDisksViaVmSpec >>>>>>>>>>>>>> >>>>>>>>>>>>>> VirtualMachineManagerImpl: pass VirtualMachineTO when migrating >>>>>>>>>>>>>> so that we can see which disks belong to the VM and what storage >>>>>>>>>>>>>> pools/adaptors should be used >>>>>>>>>>>>>> >>>>>>>>>>>>>> LibvirtComputingResource: add calls KVMStoragePoolManager's >>>>>>>>>>>>>> connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec >>>>>>>>>>>>>> calls where appropriate (when starting a vm, migrating a vm). >>>>>>>>>>>>>> Ensure that we create 'raw' format XML disk definitions when the >>>>>>>>>>>>>> storage format is RAW. Move cleanupDisk logic to storage >>>>>>>>>>>>>> adaptors so that each adaptor type can clean up its disks in is >>>>>>>>>>>>>> own way. >>>>>>>>>>>>>> >>>>>>>>>>>>>> KVMStoragePoolManager: add connectPhysicalDisk, >>>>>>>>>>>>>> disconnectPhysicalDisk, connectPhysicalDiskViaVmSpec, >>>>>>>>>>>>>> disconnectPhysicalDiskViaVmSpec, disconnectPhysicalDiskByPath. >>>>>>>>>>>>>> These all call the specific StorageAdaptor's >>>>>>>>>>>>>> connectPhysicalDisk, disconnectPhysicalDisk, or >>>>>>>>>>>>>> disconnectPhysicalDiskByPath calls. >>>>>>>>>>>>>> >>>>>>>>>>>>>> KVMStorageProcessor: Call >>>>>>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk on the storage >>>>>>>>>>>>>> adaptor. Whether or not this is implemented is up to the storage >>>>>>>>>>>>>> adaptor. >>>>>>>>>>>>>> >>>>>>>>>>>>>> LibvirtStorageAdaptor: implement dummy >>>>>>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk, move cleanupDisk >>>>>>>>>>>>>> logic from LibvirtComputingResource to >>>>>>>>>>>>>> disconnectPhysicalDiskByPath >>>>>>>>>>>>>> >>>>>>>>>>>>>> StorageAdaptor: define >>>>>>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk/disconnectPhysicalDiskByPath >>>>>>>>>>>>>> in the interface >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Testing >>>>>>>>>>>>>> >>>>>>>>>>>>>> Basic testing with my storage adaptor >>>>>>>>>>>>>> >>>>>>>>>>>>>> Diffs >>>>>>>>>>>>>> >>>>>>>>>>>>>> - core/src/com/cloud/agent/api/MigrateCommand.java >>>>>>>>>>>>>> (5042b8c) >>>>>>>>>>>>>> - >>>>>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java >>>>>>>>>>>>>> (3ee811f) >>>>>>>>>>>>>> - >>>>>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java >>>>>>>>>>>>>> (e09c9ba) >>>>>>>>>>>>>> - >>>>>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java >>>>>>>>>>>>>> (c69f9b0) >>>>>>>>>>>>>> - >>>>>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java >>>>>>>>>>>>>> (123a9f1) >>>>>>>>>>>>>> - >>>>>>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java >>>>>>>>>>>>>> (4956d8d) >>>>>>>>>>>>>> - server/src/com/cloud/vm/VirtualMachineManagerImpl.java >>>>>>>>>>>>>> (d46bbb0) >>>>>>>>>>>>>> >>>>>>>>>>>>>> View Diff <https://reviews.apache.org/r/14381/diff/> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>> Advancing the way the world uses the >>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>>>>>> *™* >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>> Advancing the way the world uses the >>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>>>>> *™* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Mike Tutkowski* >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>> o: 303.746.7302 >>>>>>>>> Advancing the way the world uses the >>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>>> *™* >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Mike Tutkowski* >>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>> o: 303.746.7302 >>>>>>>> Advancing the way the world uses the >>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>> *™* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Mike Tutkowski* >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> e: mike.tutkow...@solidfire.com >>>>>>> o: 303.746.7302 >>>>>>> Advancing the way the world uses the >>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>> *™* >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the >>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>> *™* >>>>> >>>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the >>> cloud<http://solidfire.com/solution/overview/?video=play> >>> *™* >>> >> > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™*
