Hey Marcus, Maybe you could give me a better idea of what the "flow" is when adding a KVM host.
It looks like we SSH into the potential KVM host and execute a startup script (giving it necessary info about the cloud and the management server it should talk to). After this, is the Java VM started? After a reboot, I assume the JVM is started automatically? How do you debug your KVM-side Java code? Been looking through the logs and nothing obvious sticks out. I will have another look. Thanks On Mon, Sep 23, 2013 at 2:15 PM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > Hey Marcus, > > I've been investigating my issue with not being able to add a KVM host to > CS. > > For what it's worth, this comes back successful: > > SSHCmdHelper.sshExecuteCmd(sshConnection, "cloudstack-setup-agent " + > parameters, 3); > > This is what the command looks like: > > cloudstack-setup-agent -m 192.168.233.1 -z 1 -p 1 -c 1 -g > 6b4aa1c2-2ac9-3c60-aabe-704aed40c684 -a --pubNic=cloudbr0 --prvNic=cloudbr0 > --guestNic=cloudbr0 > > The problem is this method in LibvirtServerDiscoverer never finds a > matching host in the DB: > > waitForHostConnect(long dcId, long podId, long clusterId, String guid) > > I assume once the KVM host is up and running that it's supposed to call > into the CS MS so the DB can be updated as such? > > If so, the problem must be on the KVM side. > > I did run this again (from the KVM host) to see if the connection was in > place: > > mtutkowski@ubuntu:~$ telnet 192.168.233.1 8250 > > Trying 192.168.233.1... > > Connected to 192.168.233.1. > > Escape character is '^]'. > So that looks good. > > I turned on more info in the debug log, but nothing obvious jumps out as > of yet. > > If you have any thoughts on this, please shoot them my way. :) > > Thanks! > > > On Sun, Sep 22, 2013 at 12:11 AM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > >> First step is for me to get this working for KVM, though. :) >> >> Once I do that, I can perhaps make modifications to the storage framework >> and hypervisor plug-ins to refactor the logic and such. >> >> >> On Sun, Sep 22, 2013 at 12:09 AM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >>> Same would work for KVM. >>> >>> If CreateCommand and DestroyCommand were called at the appropriate times >>> by the storage framework, I could move my connect and disconnect logic out >>> of the attach/detach logic. >>> >>> >>> On Sun, Sep 22, 2013 at 12:08 AM, Mike Tutkowski < >>> mike.tutkow...@solidfire.com> wrote: >>> >>>> Conversely, if the storage framework called the DestroyCommand for >>>> managed storage after the DetachCommand, then I could have had my remove >>>> SR/datastore logic placed in the DestroyCommand handling rather than in the >>>> DetachCommand handling. >>>> >>>> >>>> On Sun, Sep 22, 2013 at 12:06 AM, Mike Tutkowski < >>>> mike.tutkow...@solidfire.com> wrote: >>>> >>>>> Edison's plug-in calls the CreateCommand. Mine does not. >>>>> >>>>> The initial approach that was discussed during 4.2 was for me to >>>>> modify the attach/detach logic only in the XenServer and VMware hypervisor >>>>> plug-ins. >>>>> >>>>> Now that I think about it more, though, I kind of would have liked to >>>>> have the storage framework send a CreateCommand to the hypervisor before >>>>> sending the AttachCommand if the storage in question was managed. >>>>> >>>>> Then I could have created my SR/datastore in the CreateCommand and the >>>>> AttachCommand would have had the SR/datastore that it was always expecting >>>>> (and I wouldn't have had to create the SR/datastore in the AttachCommand). >>>>> >>>>> >>>>> On Sat, Sep 21, 2013 at 11:56 PM, Marcus Sorensen <shadow...@gmail.com >>>>> > wrote: >>>>> >>>>>> Yeah, I think it probably is as well, but I figured you'd be in a >>>>>> better position to tell. >>>>>> >>>>>> I see that copyAsync is unsupported in your current 4.2 driver, does >>>>>> that mean that there's no template support? Or is it some other call >>>>>> that does templating now? I'm still getting up to speed on all of the >>>>>> 4.2 changes. I was just looking at CreateCommand in >>>>>> LibvirtComputingResource, since that's the only place >>>>>> createPhysicalDisk is called, and it occurred to me that CreateCommand >>>>>> might be skipped altogether when utilizing storage plugins. >>>>>> >>>>>> On Sat, Sep 21, 2013 at 11:38 PM, Mike Tutkowski >>>>>> <mike.tutkow...@solidfire.com> wrote: >>>>>> > That's an interesting comment, Marcus. >>>>>> > >>>>>> > It was my intent that it should work with any CloudStack "managed" >>>>>> storage >>>>>> > that uses an iSCSI target. Even though I'm using CHAP, I wrote the >>>>>> code so >>>>>> > CHAP didn't have to be used. >>>>>> > >>>>>> > As I'm doing my testing, I can try to think about whether it is >>>>>> generic >>>>>> > enough to keep those names or not. >>>>>> > >>>>>> > My expectation is that it is generic enough. >>>>>> > >>>>>> > >>>>>> > On Sat, Sep 21, 2013 at 11:32 PM, Marcus Sorensen < >>>>>> shadow...@gmail.com>wrote: >>>>>> > >>>>>> >> I added a comment to your diff. In general I think it looks good, >>>>>> >> though I obviously can't vouch for whether or not it will work. One >>>>>> >> thing I do have reservations about is the adaptor/pool naming. If >>>>>> you >>>>>> >> think the code is generic enough that it will work for anyone who >>>>>> does >>>>>> >> an iscsi LUN-per-volume plugin, then it's OK, but if there's >>>>>> anything >>>>>> >> about it that's specific to YOUR iscsi target or how it likes to be >>>>>> >> treated then I'd say that they should be named something less >>>>>> generic >>>>>> >> than iScsiAdmStorage. >>>>>> >> >>>>>> >> On Sat, Sep 21, 2013 at 7:23 PM, Mike Tutkowski >>>>>> >> <mike.tutkow...@solidfire.com> wrote: >>>>>> >> > Great - thanks! >>>>>> >> > >>>>>> >> > Just to give you an overview of what my code does (for when you >>>>>> get a >>>>>> >> > chance to review it): >>>>>> >> > >>>>>> >> > SolidFireHostListener is registered in >>>>>> SolidfirePrimaryDataStoreProvider. >>>>>> >> > Its hostConnect method is invoked when a host connects with the >>>>>> CS MS. If >>>>>> >> > the host is running KVM, the listener sends a >>>>>> ModifyStoragePoolCommand to >>>>>> >> > the host. This logic was based off of DefaultHostListener. >>>>>> >> > >>>>>> >> > The handling of ModifyStoragePoolCommand is unchanged. It invokes >>>>>> >> > createStoragePool on the KVMStoragePoolManager. The >>>>>> KVMStoragePoolManager >>>>>> >> > asks for an adaptor and finds my new one: iScsiAdmStorageAdaptor >>>>>> (which >>>>>> >> was >>>>>> >> > registered in the constructor for KVMStoragePoolManager under >>>>>> the key of >>>>>> >> > StoragePoolType.Iscsi.toString()). >>>>>> >> > >>>>>> >> > iScsiAdmStorageAdaptor.createStoragePool just makes an instance >>>>>> of >>>>>> >> > iScsiAdmStoragePool, adds it to a map, and returns the pointer >>>>>> to the >>>>>> >> > iScsiAdmStoragePool object. The key of the map is the UUID of >>>>>> the storage >>>>>> >> > pool. >>>>>> >> > >>>>>> >> > When a volume is attached, createPhysicalDisk is invoked for >>>>>> managed >>>>>> >> > storage rather than getPhysicalDisk. createPhysicalDisk uses >>>>>> iscsiadm to >>>>>> >> > establish the iSCSI connection to the volume on the SAN and a >>>>>> >> > KVMPhysicalDisk is returned to be used in the attach logic that >>>>>> follows. >>>>>> >> > >>>>>> >> > When a volume is detached, getPhysicalDisk is invoked with the >>>>>> IQN of the >>>>>> >> > volume if the storage pool in question is managed storage. >>>>>> Otherwise, the >>>>>> >> > normal vol.getPath() is used. >>>>>> iScsiAdmStorageAdaptor.getPhysicalDisk just >>>>>> >> > returns a new instance of KVMPhysicalDisk to be used in the >>>>>> detach logic. >>>>>> >> > >>>>>> >> > Once the volume has been detached, >>>>>> iScsiAdmStoragePool.deletePhysicalDisk >>>>>> >> > is invoked if the storage pool is managed. deletePhysicalDisk >>>>>> removes the >>>>>> >> > iSCSI connection to the volume using iscsiadm. >>>>>> >> > >>>>>> >> > >>>>>> >> > On Sat, Sep 21, 2013 at 5:46 PM, Marcus Sorensen < >>>>>> shadow...@gmail.com >>>>>> >> >wrote: >>>>>> >> > >>>>>> >> >> Its the log4j properties file in /etc/cloudstack/agent change >>>>>> all INFO >>>>>> >> to >>>>>> >> >> DEBUG. I imagine the agent just isn't starting, you can tail >>>>>> the log >>>>>> >> when >>>>>> >> >> you try to start the service, or maybe it will spit something >>>>>> out into >>>>>> >> one >>>>>> >> >> of the other files in /var/log/cloudstack/agent >>>>>> >> >> On Sep 21, 2013 5:19 PM, "Mike Tutkowski" < >>>>>> mike.tutkow...@solidfire.com >>>>>> >> > >>>>>> >> >> wrote: >>>>>> >> >> >>>>>> >> >> > This is how I've been trying to query for the status of the >>>>>> service (I >>>>>> >> >> > assume it could be started this way, as well, by changing >>>>>> "status" to >>>>>> >> >> > "start" or "restart"?): >>>>>> >> >> > >>>>>> >> >> > mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo >>>>>> /usr/sbin/service >>>>>> >> >> > cloudstack-agent status >>>>>> >> >> > >>>>>> >> >> > I get this back: >>>>>> >> >> > >>>>>> >> >> > Failed to execute: * could not access PID file for >>>>>> cloudstack-agent >>>>>> >> >> > >>>>>> >> >> > I've made a bunch of code changes recently, though, so I >>>>>> think I'm >>>>>> >> going >>>>>> >> >> to >>>>>> >> >> > rebuild and redeploy everything. >>>>>> >> >> > >>>>>> >> >> > The debug info sounds helpful. Where can I set enable.debug? >>>>>> >> >> > >>>>>> >> >> > Thanks, Marcus! >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > On Sat, Sep 21, 2013 at 4:11 PM, Marcus Sorensen < >>>>>> shadow...@gmail.com >>>>>> >> >> > >wrote: >>>>>> >> >> > >>>>>> >> >> > > OK, will check it out in the next few days. As mentioned, >>>>>> you can >>>>>> >> set >>>>>> >> >> up >>>>>> >> >> > > your Ubuntu vm as the management server as well if all else >>>>>> fails. >>>>>> >> If >>>>>> >> >> > you >>>>>> >> >> > > can get to the mgmt server on 8250 from the KVM host, then >>>>>> you need >>>>>> >> to >>>>>> >> >> > > enable.debug on the agent. It won't run without complaining >>>>>> loudly >>>>>> >> if >>>>>> >> >> it >>>>>> >> >> > > can't get to the mgmt server, and I didn't see that in your >>>>>> agent >>>>>> >> log, >>>>>> >> >> so >>>>>> >> >> > > perhaps its not running. I assume you know how to >>>>>> stop/start the >>>>>> >> agent >>>>>> >> >> on >>>>>> >> >> > > KVM via 'service cloud stacks agent'. >>>>>> >> >> > > On Sep 21, 2013 3:02 PM, "Mike Tutkowski" < >>>>>> >> >> mike.tutkow...@solidfire.com> >>>>>> >> >> > > wrote: >>>>>> >> >> > > >>>>>> >> >> > > > Hey Marcus, >>>>>> >> >> > > > >>>>>> >> >> > > > I haven't yet been able to test my new code, but I >>>>>> thought you >>>>>> >> would >>>>>> >> >> > be a >>>>>> >> >> > > > good person to ask to review it: >>>>>> >> >> > > > >>>>>> >> >> > > > >>>>>> >> >> > > > >>>>>> >> >> > > >>>>>> >> >> > >>>>>> >> >> >>>>>> >> >>>>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/ea74b312a8a36801994500407fd54f0cdda55e37 >>>>>> >> >> > > > >>>>>> >> >> > > > All it is supposed to do is attach and detach a data disk >>>>>> (that >>>>>> >> has >>>>>> >> >> > > > guaranteed IOPS) with KVM as the hypervisor. The data disk >>>>>> >> happens to >>>>>> >> >> > be >>>>>> >> >> > > > from SolidFire-backed storage - where we have a 1:1 >>>>>> mapping >>>>>> >> between a >>>>>> >> >> > > > CloudStack volume and a data disk. >>>>>> >> >> > > > >>>>>> >> >> > > > There is no support for hypervisor snapshots or stuff >>>>>> like that >>>>>> >> >> > (likely a >>>>>> >> >> > > > future release)...just attaching and detaching a data >>>>>> disk in 4.3. >>>>>> >> >> > > > >>>>>> >> >> > > > Thanks! >>>>>> >> >> > > > >>>>>> >> >> > > > >>>>>> >> >> > > > On Fri, Sep 20, 2013 at 11:05 PM, Mike Tutkowski < >>>>>> >> >> > > > mike.tutkow...@solidfire.com> wrote: >>>>>> >> >> > > > >>>>>> >> >> > > > > When I re-deployed the DEBs, I didn't remove >>>>>> cloudstack-agent >>>>>> >> >> first. >>>>>> >> >> > > > Would >>>>>> >> >> > > > > that be a problem? I just did a sudo apt-get install >>>>>> >> >> > cloudstack-agent. >>>>>> >> >> > > > > >>>>>> >> >> > > > > >>>>>> >> >> > > > > On Fri, Sep 20, 2013 at 11:03 PM, Mike Tutkowski < >>>>>> >> >> > > > > mike.tutkow...@solidfire.com> wrote: >>>>>> >> >> > > > > >>>>>> >> >> > > > >> I get the same error running the command manually: >>>>>> >> >> > > > >> >>>>>> >> >> > > > >> mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo >>>>>> >> /usr/sbin/service >>>>>> >> >> > > > >> cloudstack-agent status >>>>>> >> >> > > > >> * could not access PID file for cloudstack-agent >>>>>> >> >> > > > >> >>>>>> >> >> > > > >> >>>>>> >> >> > > > >> On Fri, Sep 20, 2013 at 11:00 PM, Mike Tutkowski < >>>>>> >> >> > > > >> mike.tutkow...@solidfire.com> wrote: >>>>>> >> >> > > > >> >>>>>> >> >> > > > >>> agent.log looks OK to me: >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,010 INFO [cloud.agent.AgentShell] >>>>>> >> >> (main:null) >>>>>> >> >> > > > Agent >>>>>> >> >> > > > >>> started >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,011 INFO [cloud.agent.AgentShell] >>>>>> >> >> (main:null) >>>>>> >> >> > > > >>> Implementation Version is 4.3.0-SNAPSHOT >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,015 INFO [cloud.agent.AgentShell] >>>>>> >> >> (main:null) >>>>>> >> >> > > > >>> agent.properties found at >>>>>> >> /etc/cloudstack/agent/agent.properties >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,023 INFO [cloud.agent.AgentShell] >>>>>> >> >> (main:null) >>>>>> >> >> > > > >>> Defaulting to using properties file for storage >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,024 INFO [cloud.agent.AgentShell] >>>>>> >> >> (main:null) >>>>>> >> >> > > > >>> Defaulting to the constant time backoff algorithm >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,029 INFO [cloud.utils.LogUtils] >>>>>> >> (main:null) >>>>>> >> >> > > log4j >>>>>> >> >> > > > >>> configuration found at >>>>>> /etc/cloudstack/agent/log4j-cloud.xml >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,163 INFO [cloud.agent.Agent] >>>>>> (main:null) >>>>>> >> id >>>>>> >> >> > is 3 >>>>>> >> >> > > > >>> 2013-09-20 19:35:39,197 INFO >>>>>> >> >> > > > >>> [resource.virtualnetwork.VirtualRoutingResource] >>>>>> (main:null) >>>>>> >> >> > > > >>> VirtualRoutingResource _scriptDir to use: >>>>>> >> >> scripts/network/domr/kvm >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> However, I wasn't aware that setup.log was important. >>>>>> This >>>>>> >> seems >>>>>> >> >> to >>>>>> >> >> > > be >>>>>> >> >> > > > a >>>>>> >> >> > > > >>> problem, but I'm not sure what it might indicate: >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service >>>>>> cloudstack-agent >>>>>> >> status >>>>>> >> >> > > > >>> DEBUG:root:Failed to execute: * could not access PID >>>>>> file for >>>>>> >> >> > > > >>> cloudstack-agent >>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service >>>>>> cloudstack-agent >>>>>> >> start >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> On Fri, Sep 20, 2013 at 10:53 PM, Marcus Sorensen < >>>>>> >> >> > > shadow...@gmail.com >>>>>> >> >> > > > >wrote: >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>>> Sorry, I saw that in the log, I thought it was the >>>>>> agent log >>>>>> >> for >>>>>> >> >> > > some >>>>>> >> >> > > > >>>> reason. Is the agent started? That might be the >>>>>> place to >>>>>> >> look. >>>>>> >> >> > There >>>>>> >> >> > > > is >>>>>> >> >> > > > >>>> an >>>>>> >> >> > > > >>>> agent log for the agent and one for the setup when >>>>>> it adds >>>>>> >> the >>>>>> >> >> > host, >>>>>> >> >> > > > >>>> both >>>>>> >> >> > > > >>>> in /var/log >>>>>> >> >> > > > >>>> On Sep 20, 2013 10:42 PM, "Mike Tutkowski" < >>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com> >>>>>> >> >> > > > >>>> wrote: >>>>>> >> >> > > > >>>> >>>>>> >> >> > > > >>>> > Is it saying that the MS is at the IP address or >>>>>> the KVM >>>>>> >> host? >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > The KVM host is at 192.168.233.10. >>>>>> >> >> > > > >>>> > The MS host is at 192.168.233.1. >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > I see this for my host Global Settings parameter: >>>>>> >> >> > > > >>>> > hostThe ip address of management >>>>>> server192.168.233.1 >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > /etc/cloudstack/agent/agent.properties has a >>>>>> >> >> host=192.168.233.1 >>>>>> >> >> > > > value. >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > On Fri, Sep 20, 2013 at 10:21 PM, Marcus Sorensen < >>>>>> >> >> > > > >>>> shadow...@gmail.com >>>>>> >> >> > > > >>>> > >wrote: >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > > The log says your mgmt server is 192.168.233.10? >>>>>> But you >>>>>> >> >> tried >>>>>> >> >> > > to >>>>>> >> >> > > > >>>> telnet >>>>>> >> >> > > > >>>> > to >>>>>> >> >> > > > >>>> > > 192.168.233.1? It might be enough to change that >>>>>> in >>>>>> >> >> > > > >>>> > > /etc/cloudstack/agent/agent.properties, but you >>>>>> may want >>>>>> >> to >>>>>> >> >> > edit >>>>>> >> >> > > > the >>>>>> >> >> > > > >>>> > config >>>>>> >> >> > > > >>>> > > as well to tell it the real ms IP. >>>>>> >> >> > > > >>>> > > On Sep 20, 2013 10:12 PM, "Mike Tutkowski" < >>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com >>>>>> >> >> > > > >>>> > > >>>>>> >> >> > > > >>>> > > wrote: >>>>>> >> >> > > > >>>> > > >>>>>> >> >> > > > >>>> > > > Here's what my /etc/network/interfaces file >>>>>> looks >>>>>> >> like, if >>>>>> >> >> > > that >>>>>> >> >> > > > >>>> is of >>>>>> >> >> > > > >>>> > > > interest (the 192.168.233.0 network is the NAT >>>>>> network >>>>>> >> >> > VMware >>>>>> >> >> > > > >>>> Fusion >>>>>> >> >> > > > >>>> > set >>>>>> >> >> > > > >>>> > > > up): >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > auto lo >>>>>> >> >> > > > >>>> > > > iface lo inet loopback >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > auto eth0 >>>>>> >> >> > > > >>>> > > > iface eth0 inet manual >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > auto cloudbr0 >>>>>> >> >> > > > >>>> > > > iface cloudbr0 inet static >>>>>> >> >> > > > >>>> > > > address 192.168.233.10 >>>>>> >> >> > > > >>>> > > > netmask 255.255.255.0 >>>>>> >> >> > > > >>>> > > > network 192.168.233.0 >>>>>> >> >> > > > >>>> > > > broadcast 192.168.233.255 >>>>>> >> >> > > > >>>> > > > dns-nameservers 8.8.8.8 >>>>>> >> >> > > > >>>> > > > bridge_ports eth0 >>>>>> >> >> > > > >>>> > > > bridge_fd 5 >>>>>> >> >> > > > >>>> > > > bridge_stp off >>>>>> >> >> > > > >>>> > > > bridge_maxwait 1 >>>>>> >> >> > > > >>>> > > > post-up route add default gw 192.168.233.2 >>>>>> metric 1 >>>>>> >> >> > > > >>>> > > > pre-down route del default gw 192.168.233.2 >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > On Fri, Sep 20, 2013 at 10:08 PM, Mike >>>>>> Tutkowski < >>>>>> >> >> > > > >>>> > > > mike.tutkow...@solidfire.com> wrote: >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > > You appear to be correct. This is from the >>>>>> MS log >>>>>> >> >> (below). >>>>>> >> >> > > > >>>> Discovery >>>>>> >> >> > > > >>>> > > > timed >>>>>> >> >> > > > >>>> > > > > out. >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > I'm not sure why this would be. My network >>>>>> settings >>>>>> >> >> > > shouldn't >>>>>> >> >> > > > >>>> have >>>>>> >> >> > > > >>>> > > > changed >>>>>> >> >> > > > >>>> > > > > since the last time I tried this. >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > I am able to ping the KVM host from the MS >>>>>> host and >>>>>> >> vice >>>>>> >> >> > > > versa. >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > I'm even able to manually kick off a VM on >>>>>> the KVM >>>>>> >> host >>>>>> >> >> > and >>>>>> >> >> > > > >>>> ping from >>>>>> >> >> > > > >>>> > > it >>>>>> >> >> > > > >>>> > > > > to the MS host. >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > I am using NAT from my Mac OS X host (also >>>>>> running >>>>>> >> the >>>>>> >> >> CS >>>>>> >> >> > > MS) >>>>>> >> >> > > > >>>> to the >>>>>> >> >> > > > >>>> > VM >>>>>> >> >> > > > >>>> > > > > running KVM (VMware Fusion). >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,141 DEBUG >>>>>> >> >> > > > >>>> [c.c.h.k.d.LibvirtServerDiscoverer] >>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3:ctx-6b28dc48) >>>>>> Timeout, >>>>>> >> to >>>>>> >> >> > wait >>>>>> >> >> > > > for >>>>>> >> >> > > > >>>> the >>>>>> >> >> > > > >>>> > > host >>>>>> >> >> > > > >>>> > > > > connecting to mgt svr, assuming it is failed >>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,144 WARN >>>>>> >> >> [c.c.r.ResourceManagerImpl] >>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3:ctx-6b28dc48) >>>>>> Unable to >>>>>> >> >> find >>>>>> >> >> > > the >>>>>> >> >> > > > >>>> server >>>>>> >> >> > > > >>>> > > > > resources at http://192.168.233.10 >>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,145 INFO >>>>>> >> >> > [c.c.u.e.CSExceptionErrorCode] >>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3:ctx-6b28dc48) >>>>>> Could not >>>>>> >> >> find >>>>>> >> >> > > > >>>> exception: >>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException in >>>>>> error code >>>>>> >> >> list >>>>>> >> >> > > for >>>>>> >> >> > > > >>>> > > exceptions >>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,147 WARN >>>>>> >> [o.a.c.a.c.a.h.AddHostCmd] >>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3:ctx-6b28dc48) >>>>>> Exception: >>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException: >>>>>> Unable to add >>>>>> >> >> the >>>>>> >> >> > > host >>>>>> >> >> > > > >>>> > > > > at >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> >>>>>> >> >> > > > >>>>>> >> >> > > >>>>>> >> >> > >>>>>> >> >> >>>>>> >> >>>>>> com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:778) >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > I do seem to be able to telnet in from my >>>>>> KVM host to >>>>>> >> >> the >>>>>> >> >> > MS >>>>>> >> >> > > > >>>> host's >>>>>> >> >> > > > >>>> > > 8250 >>>>>> >> >> > > > >>>> > > > > port: >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > > mtutkowski@ubuntu:~$ telnet 192.168.233.1 >>>>>> 8250 >>>>>> >> >> > > > >>>> > > > > Trying 192.168.233.1... >>>>>> >> >> > > > >>>> > > > > Connected to 192.168.233.1. >>>>>> >> >> > > > >>>> > > > > Escape character is '^]'. >>>>>> >> >> > > > >>>> > > > > >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > > -- >>>>>> >> >> > > > >>>> > > > *Mike Tutkowski* >>>>>> >> >> > > > >>>> > > > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > > > >>>> > > > e: mike.tutkow...@solidfire.com >>>>>> >> >> > > > >>>> > > > o: 303.746.7302 >>>>>> >> >> > > > >>>> > > > Advancing the way the world uses the >>>>>> >> >> > > > >>>> > > > cloud< >>>>>> >> http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > > > >>>> > > > *™* >>>>>> >> >> > > > >>>> > > > >>>>>> >> >> > > > >>>> > > >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> > -- >>>>>> >> >> > > > >>>> > *Mike Tutkowski* >>>>>> >> >> > > > >>>> > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > > > >>>> > e: mike.tutkow...@solidfire.com >>>>>> >> >> > > > >>>> > o: 303.746.7302 >>>>>> >> >> > > > >>>> > Advancing the way the world uses the >>>>>> >> >> > > > >>>> > cloud< >>>>>> http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > > > >>>> > *™* >>>>>> >> >> > > > >>>> > >>>>>> >> >> > > > >>>> >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >>> -- >>>>>> >> >> > > > >>> *Mike Tutkowski* >>>>>> >> >> > > > >>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > > > >>> e: mike.tutkow...@solidfire.com >>>>>> >> >> > > > >>> o: 303.746.7302 >>>>>> >> >> > > > >>> Advancing the way the world uses the cloud< >>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > > > >>> *™* >>>>>> >> >> > > > >>> >>>>>> >> >> > > > >> >>>>>> >> >> > > > >> >>>>>> >> >> > > > >> >>>>>> >> >> > > > >> -- >>>>>> >> >> > > > >> *Mike Tutkowski* >>>>>> >> >> > > > >> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > > > >> e: mike.tutkow...@solidfire.com >>>>>> >> >> > > > >> o: 303.746.7302 >>>>>> >> >> > > > >> Advancing the way the world uses the cloud< >>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > > > >> *™* >>>>>> >> >> > > > >> >>>>>> >> >> > > > > >>>>>> >> >> > > > > >>>>>> >> >> > > > > >>>>>> >> >> > > > > -- >>>>>> >> >> > > > > *Mike Tutkowski* >>>>>> >> >> > > > > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > > > > e: mike.tutkow...@solidfire.com >>>>>> >> >> > > > > o: 303.746.7302 >>>>>> >> >> > > > > Advancing the way the world uses the cloud< >>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > > > > *™* >>>>>> >> >> > > > > >>>>>> >> >> > > > >>>>>> >> >> > > > >>>>>> >> >> > > > >>>>>> >> >> > > > -- >>>>>> >> >> > > > *Mike Tutkowski* >>>>>> >> >> > > > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > > > e: mike.tutkow...@solidfire.com >>>>>> >> >> > > > o: 303.746.7302 >>>>>> >> >> > > > Advancing the way the world uses the >>>>>> >> >> > > > cloud<http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > > > *™* >>>>>> >> >> > > > >>>>>> >> >> > > >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > -- >>>>>> >> >> > *Mike Tutkowski* >>>>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> >> > e: mike.tutkow...@solidfire.com >>>>>> >> >> > o: 303.746.7302 >>>>>> >> >> > Advancing the way the world uses the >>>>>> >> >> > cloud<http://solidfire.com/solution/overview/?video=play> >>>>>> >> >> > *™* >>>>>> >> >> > >>>>>> >> >> >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > -- >>>>>> >> > *Mike Tutkowski* >>>>>> >> > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> >> > e: mike.tutkow...@solidfire.com >>>>>> >> > o: 303.746.7302 >>>>>> >> > Advancing the way the world uses the >>>>>> >> > cloud<http://solidfire.com/solution/overview/?video=play> >>>>>> >> > *™* >>>>>> >> >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > *Mike Tutkowski* >>>>>> > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> > e: mike.tutkow...@solidfire.com >>>>>> > o: 303.746.7302 >>>>>> > Advancing the way the world uses the >>>>>> > cloud<http://solidfire.com/solution/overview/?video=play> >>>>>> > *™* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the >>>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>>> *™* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the >>>> cloud<http://solidfire.com/solution/overview/?video=play> >>>> *™* >>>> >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the >>> cloud<http://solidfire.com/solution/overview/?video=play> >>> *™* >>> >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the >> cloud<http://solidfire.com/solution/overview/?video=play> >> *™* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*