You may just have to wait a bit. That means the VMS are started but the services aren't up. Usually takes 5-10 min to get both up in my fusion vm. If that doesn't get better you may have to log into the system VMS and see what's going on. On Oct 8, 2013 7:25 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> wrote:
> Doh! I didn't restart the necessary service after exporting my NFS share. > > The system VMs come up now; however, SSVM has a dash (-) for its Agent > State and the CS MS console keeps printing out the following: > > INFO [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-3:ctx-018adc41) > No running ssvm is found, so command will be sent to LocalHostEndPoint > > > On Tue, Oct 8, 2013 at 4:35 PM, Marcus Sorensen <shadow...@gmail.com>wrote: > >> Can you mount the secondary storage from your KVM host? >> On Oct 8, 2013 4:01 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> >> wrote: >> >>> Although the host is added to KVM, I do see the following issues in the >>> CS MS console (any thoughts on this?): >>> >>> WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-c1c573ee) Was unable to >>> find lock for the key template_spool_ref2 and thread id 2049868806 >>> INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-c1c573ee) >>> Unable to contact resource. >>> com.cloud.exception.StorageUnavailableException: Resource >>> [StoragePool:1] is unreachable: Unable to create >>> Vol[1|vm=1|ROOT]:com.cloud.utils.exception.CloudRuntimeException: >>> org.libvirt.LibvirtException: internal error Child process (/bin/mount >>> 192.168.233.10:/mnt/secondary/template/tmpl/1/3 >>> /mnt/334b3c4e-764b-362a-be2c-ebe8c490d0a9) status unexpected: exit status 32 >>> at >>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1027) >>> at >>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1069) >>> at >>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:830) >>> at >>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:649) >>> at >>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261) >>> at >>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:693) >>> at >>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265) >>> at >>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123) >>> at >>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) >>> at >>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101) >>> at >>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) >>> at >>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78) >>> at >>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:724) >>> >>> >>> On Tue, Oct 8, 2013 at 3:58 PM, Mike Tutkowski < >>> mike.tutkow...@solidfire.com> wrote: >>> >>>> So...got some good news: >>>> >>>> I spent a couple hours setting up a KVM environment on Ubuntu 12.04.1 >>>> from scratch (Installing SSH, Open iSCSI, Java 7, KVM, Git, CloudStack, >>>> CloudStack DEBs, KVM system template, etc.) and I can now add this KVM host >>>> to CloudStack (on a related note, no errors in agent.err either). >>>> >>>> I have no idea what is messed up with my old KVM install on Ubuntu, but >>>> the new one works. >>>> >>>> That being the case, I can close out the JIRA ticket I logged a while >>>> back and start integrating your code into mine. >>>> >>>> >>>> On Mon, Oct 7, 2013 at 7:46 PM, Mike Tutkowski < >>>> mike.tutkow...@solidfire.com> wrote: >>>> >>>>> Right...yeah, I didn't mean we'd commit to 4.2, but maybe I should >>>>> work off of 4.2 since master seems to be un-stable in this regard. >>>>> >>>>> I plan to set up a machine in the lab tomorrow with Ubuntu 12.04 from >>>>> scratch to see if it works when I start clean, but - if it doesn't - I >>>>> should just use 4.2 for development. >>>>> >>>>> >>>>> On Mon, Oct 7, 2013 at 7:05 PM, Marcus Sorensen >>>>> <shadow...@gmail.com>wrote: >>>>> >>>>>> We can't. This patch will never see 4.2. You can still start working >>>>>> on your plugin on 4.2, but the change represented by this patch can only >>>>>> go >>>>>> into master. >>>>>> On Oct 7, 2013 5:01 PM, "Mike Tutkowski" < >>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>> >>>>>>> So, now that I'm getting back to this, do you think I should just >>>>>>> try to make this work with 4.2 (like we originally talked about)? >>>>>>> >>>>>>> I updated again from master, rebuilt, redeployed DEBs and still get >>>>>>> this JNA error message: >>>>>>> >>>>>>> log4j:WARN No appenders could be found for logger >>>>>>> (org.apache.commons.httpclient.params.DefaultHttpParams). >>>>>>> log4j:WARN Please initialize the log4j system properly. >>>>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfigfor >>>>>>> more info. >>>>>>> java.lang.reflect.InvocationTargetException >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> at >>>>>>> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) >>>>>>> Caused by: java.lang.UnsatisfiedLinkError: Can't obtain >>>>>>> updateLastError method for class com.sun.jna.Native >>>>>>> at com.sun.jna.Native.initIDs(Native Method) >>>>>>> at com.sun.jna.Native.<clinit>(Native.java:139) >>>>>>> at org.libvirt.jna.Libvirt.<clinit>(Unknown Source) >>>>>>> at org.libvirt.Library.<clinit>(Unknown Source) >>>>>>> at org.libvirt.Connect.<init>(Unknown Source) >>>>>>> at >>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:44) >>>>>>> at >>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:37) >>>>>>> at >>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:733) >>>>>>> at com.cloud.agent.Agent.<init>(Agent.java:161) >>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415) >>>>>>> at >>>>>>> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:370) >>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) >>>>>>> at com.cloud.agent.AgentShell.start(AgentShell.java:448) >>>>>>> ... 5 more >>>>>>> Cannot start daemon >>>>>>> Service exit with a return value of 5 >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 7, 2013 at 2:31 PM, Mike Tutkowski < >>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>> >>>>>>>> Sure, that's a good plan. >>>>>>>> >>>>>>>> I'll get to it. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 7, 2013 at 2:29 PM, Marcus Sorensen < >>>>>>>> shadow...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I know you mentioned you might need some minor changes to it, as >>>>>>>>> well as other minor changes just for master (attach volume switched >>>>>>>>> to pool >>>>>>>>> vs adapter or something). My hope was that you would be able to send >>>>>>>>> an >>>>>>>>> update that works for your plugin on master, I'll test against >>>>>>>>> existing >>>>>>>>> libvirtd storage and apply it. >>>>>>>>> On Oct 7, 2013 1:49 PM, "Mike Tutkowski" < >>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>> >>>>>>>>>> This is an automatically generated e-mail. To reply, visit: >>>>>>>>>> https://reviews.apache.org/r/14381/ >>>>>>>>>> >>>>>>>>>> This looks reasonable to me, Marcus. >>>>>>>>>> >>>>>>>>>> When do you think you might start the process of getting this into >>>>>>>>>> master? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - Mike Tutkowski >>>>>>>>>> >>>>>>>>>> On September 30th, 2013, 5:14 p.m. UTC, Marcus Sorensen wrote: >>>>>>>>>> Review request for cloudstack, edison su and Mike Tutkowski. >>>>>>>>>> By Marcus Sorensen. >>>>>>>>>> >>>>>>>>>> *Updated Sept. 30, 2013, 5:14 p.m.* >>>>>>>>>> *Repository: * cloudstack-git >>>>>>>>>> Description >>>>>>>>>> >>>>>>>>>> With custom storage plugins comes the need to prep the KVM host >>>>>>>>>> prior to utilizing the disks. e.g. an iscsi initiator needs to log >>>>>>>>>> into the target and scan for the lun before it can be used on the >>>>>>>>>> host. This patch is an example I developed against 4.2, minor >>>>>>>>>> changes may be necessary to apply to master, but I want to share >>>>>>>>>> with others who are working on storage so they can ensure it works >>>>>>>>>> for them. Please tweak as you see fit. >>>>>>>>>> >>>>>>>>>> MigrateCommand: pass vmTO object so we can see which disks/storage >>>>>>>>>> pool types belong to the vm when migrating a VM. This facilitates >>>>>>>>>> being able to call disconnectPhysicalDisksViaVmSpec >>>>>>>>>> >>>>>>>>>> VirtualMachineManagerImpl: pass VirtualMachineTO when migrating so >>>>>>>>>> that we can see which disks belong to the VM and what storage >>>>>>>>>> pools/adaptors should be used >>>>>>>>>> >>>>>>>>>> LibvirtComputingResource: add calls KVMStoragePoolManager's >>>>>>>>>> connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec >>>>>>>>>> calls where appropriate (when starting a vm, migrating a vm). Ensure >>>>>>>>>> that we create 'raw' format XML disk definitions when the storage >>>>>>>>>> format is RAW. Move cleanupDisk logic to storage adaptors so that >>>>>>>>>> each adaptor type can clean up its disks in is own way. >>>>>>>>>> >>>>>>>>>> KVMStoragePoolManager: add connectPhysicalDisk, >>>>>>>>>> disconnectPhysicalDisk, connectPhysicalDiskViaVmSpec, >>>>>>>>>> disconnectPhysicalDiskViaVmSpec, disconnectPhysicalDiskByPath. These >>>>>>>>>> all call the specific StorageAdaptor's connectPhysicalDisk, >>>>>>>>>> disconnectPhysicalDisk, or disconnectPhysicalDiskByPath calls. >>>>>>>>>> >>>>>>>>>> KVMStorageProcessor: Call connectPhysicalDisk/disconnectPhysicalDisk >>>>>>>>>> on the storage adaptor. Whether or not this is implemented is up to >>>>>>>>>> the storage adaptor. >>>>>>>>>> >>>>>>>>>> LibvirtStorageAdaptor: implement dummy >>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk, move cleanupDisk logic >>>>>>>>>> from LibvirtComputingResource to disconnectPhysicalDiskByPath >>>>>>>>>> >>>>>>>>>> StorageAdaptor: define >>>>>>>>>> connectPhysicalDisk/disconnectPhysicalDisk/disconnectPhysicalDiskByPath >>>>>>>>>> in the interface >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Testing >>>>>>>>>> >>>>>>>>>> Basic testing with my storage adaptor >>>>>>>>>> >>>>>>>>>> Diffs >>>>>>>>>> >>>>>>>>>> - core/src/com/cloud/agent/api/MigrateCommand.java (5042b8c) >>>>>>>>>> - >>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java >>>>>>>>>> (3ee811f) >>>>>>>>>> - >>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java >>>>>>>>>> (e09c9ba) >>>>>>>>>> - >>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java >>>>>>>>>> (c69f9b0) >>>>>>>>>> - >>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java >>>>>>>>>> (123a9f1) >>>>>>>>>> - >>>>>>>>>> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java >>>>>>>>>> (4956d8d) >>>>>>>>>> - server/src/com/cloud/vm/VirtualMachineManagerImpl.java >>>>>>>>>> (d46bbb0) >>>>>>>>>> >>>>>>>>>> View Diff <https://reviews.apache.org/r/14381/diff/> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Mike Tutkowski* >>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>> o: 303.746.7302 >>>>>>>> Advancing the way the world uses the >>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>>> *™* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Mike Tutkowski* >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> e: mike.tutkow...@solidfire.com >>>>>>> o: 303.746.7302 >>>>>>> Advancing the way the world uses the >>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>>>> *™* >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the >>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>> *™* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the >>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>> *™* >>>> >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the >>> cloud<http://solidfire.com/solution/overview/?video=play> >>> *™* >>> >> > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* >
