OK, so, nothing is screaming out in the logs. I did notice the following: >From setup.log:
DEBUG:root:execute:apparmor_status |grep libvirt DEBUG:root:Failed to execute: DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status DEBUG:root:Failed to execute: * could not access PID file for cloudstack-agent This is the final line in this log file: DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start This is from agent.log: 2013-09-23 15:30:55,549 DEBUG [cloud.agent.AgentShell] (main:null) Checking to see if agent.pid exists. 2013-09-23 15:30:55,655 DEBUG [cloud.utils.ProcessUtil] (main:null) Executing: bash -c echo $PPID 2013-09-23 15:30:55,742 DEBUG [cloud.utils.ProcessUtil] (main:null) Execution is successful. 2013-09-23 15:30:56,000 INFO [cloud.agent.Agent] (main:null) id is 2013-09-23 15:30:56,000 DEBUG [cloud.resource.ServerResourceBase] (main:null) Retrieving network interface: cloudbr0 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase] (main:null) Retrieving network interface: cloudbr0 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase] (main:null) Retrieving network interface: null 2013-09-23 15:30:56,017 DEBUG [cloud.resource.ServerResourceBase] (main:null) Retrieving network interface: null The following kinds of lines are repeated for a bunch of different .sh files. I think they often end up being found here: /usr/share/cloudstack-common/scripts/network/domr, so this is probably not an issue. 2013-09-23 15:30:56,111 DEBUG [utils.script.Script] (main:null) Looking for call_firewall.sh in the classpath 2013-09-23 15:30:56,112 DEBUG [utils.script.Script] (main:null) System resource: null 2013-09-23 15:30:56,113 DEBUG [utils.script.Script] (main:null) Classpath resource: null 2013-09-23 15:30:56,123 DEBUG [utils.script.Script] (main:null) Looking for call_firewall.sh Is there a log file for the Java code that I could write stuff out to and see how far we get? On Mon, Sep 23, 2013 at 3:17 PM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > Thanks, Marcus > > I've been developing on Windows for most of my time, so a bunch of these > Linux-type commands are new to me and I don't always interpret the output > correctly. Getting there. :) > > > On Mon, Sep 23, 2013 at 2:37 PM, Marcus Sorensen <shadow...@gmail.com>wrote: > >> Nope, not running. That's just your grep process. It would look like: >> >> root 24429 24428 1 14:25 ? 00:00:08 jsvc.exec -cp >> >> /usr/share/java/commons-daemon.jar:/usr/share/java/jna.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjrt-1.7. >> >> Your agent log should tell you why it failed to start if you set it in >> debug and try to start... or maybe cloudstack-agent.out if it doesn't >> get far enough (say it's missing a class or something and can't >> start). >> >> On Mon, Sep 23, 2013 at 2:33 PM, Mike Tutkowski >> <mike.tutkow...@solidfire.com> wrote: >> > Looks like it's running, though: >> > >> > mtutkowski@ubuntu:~$ ps -ef | grep jsvc >> > 1000 7097 7013 0 14:32 pts/1 00:00:00 grep --color=auto jsvc >> > >> > >> > >> > On Mon, Sep 23, 2013 at 2:31 PM, Mike Tutkowski < >> > mike.tutkow...@solidfire.com> wrote: >> > >> >> Hey Marcus, >> >> >> >> Maybe you could give me a better idea of what the "flow" is when >> adding a >> >> KVM host. >> >> >> >> It looks like we SSH into the potential KVM host and execute a startup >> >> script (giving it necessary info about the cloud and the management >> server >> >> it should talk to). >> >> >> >> After this, is the Java VM started? >> >> >> >> After a reboot, I assume the JVM is started automatically? >> >> >> >> How do you debug your KVM-side Java code? >> >> >> >> Been looking through the logs and nothing obvious sticks out. I will >> have >> >> another look. >> >> >> >> Thanks >> >> >> >> >> >> On Mon, Sep 23, 2013 at 2:15 PM, Mike Tutkowski < >> >> mike.tutkow...@solidfire.com> wrote: >> >> >> >>> Hey Marcus, >> >>> >> >>> I've been investigating my issue with not being able to add a KVM >> host to >> >>> CS. >> >>> >> >>> For what it's worth, this comes back successful: >> >>> >> >>> SSHCmdHelper.sshExecuteCmd(sshConnection, "cloudstack-setup-agent " + >> >>> parameters, 3); >> >>> >> >>> This is what the command looks like: >> >>> >> >>> cloudstack-setup-agent -m 192.168.233.1 -z 1 -p 1 -c 1 -g >> >>> 6b4aa1c2-2ac9-3c60-aabe-704aed40c684 -a --pubNic=cloudbr0 >> --prvNic=cloudbr0 >> >>> --guestNic=cloudbr0 >> >>> >> >>> The problem is this method in LibvirtServerDiscoverer never finds a >> >>> matching host in the DB: >> >>> >> >>> waitForHostConnect(long dcId, long podId, long clusterId, String guid) >> >>> >> >>> I assume once the KVM host is up and running that it's supposed to >> call >> >>> into the CS MS so the DB can be updated as such? >> >>> >> >>> If so, the problem must be on the KVM side. >> >>> >> >>> I did run this again (from the KVM host) to see if the connection was >> in >> >>> place: >> >>> >> >>> mtutkowski@ubuntu:~$ telnet 192.168.233.1 8250 >> >>> >> >>> Trying 192.168.233.1... >> >>> >> >>> Connected to 192.168.233.1. >> >>> >> >>> Escape character is '^]'. >> >>> So that looks good. >> >>> >> >>> I turned on more info in the debug log, but nothing obvious jumps out >> as >> >>> of yet. >> >>> >> >>> If you have any thoughts on this, please shoot them my way. :) >> >>> >> >>> Thanks! >> >>> >> >>> >> >>> On Sun, Sep 22, 2013 at 12:11 AM, Mike Tutkowski < >> >>> mike.tutkow...@solidfire.com> wrote: >> >>> >> >>>> First step is for me to get this working for KVM, though. :) >> >>>> >> >>>> Once I do that, I can perhaps make modifications to the storage >> >>>> framework and hypervisor plug-ins to refactor the logic and such. >> >>>> >> >>>> >> >>>> On Sun, Sep 22, 2013 at 12:09 AM, Mike Tutkowski < >> >>>> mike.tutkow...@solidfire.com> wrote: >> >>>> >> >>>>> Same would work for KVM. >> >>>>> >> >>>>> If CreateCommand and DestroyCommand were called at the appropriate >> >>>>> times by the storage framework, I could move my connect and >> disconnect >> >>>>> logic out of the attach/detach logic. >> >>>>> >> >>>>> >> >>>>> On Sun, Sep 22, 2013 at 12:08 AM, Mike Tutkowski < >> >>>>> mike.tutkow...@solidfire.com> wrote: >> >>>>> >> >>>>>> Conversely, if the storage framework called the DestroyCommand for >> >>>>>> managed storage after the DetachCommand, then I could have had my >> remove >> >>>>>> SR/datastore logic placed in the DestroyCommand handling rather >> than in the >> >>>>>> DetachCommand handling. >> >>>>>> >> >>>>>> >> >>>>>> On Sun, Sep 22, 2013 at 12:06 AM, Mike Tutkowski < >> >>>>>> mike.tutkow...@solidfire.com> wrote: >> >>>>>> >> >>>>>>> Edison's plug-in calls the CreateCommand. Mine does not. >> >>>>>>> >> >>>>>>> The initial approach that was discussed during 4.2 was for me to >> >>>>>>> modify the attach/detach logic only in the XenServer and VMware >> hypervisor >> >>>>>>> plug-ins. >> >>>>>>> >> >>>>>>> Now that I think about it more, though, I kind of would have >> liked to >> >>>>>>> have the storage framework send a CreateCommand to the hypervisor >> before >> >>>>>>> sending the AttachCommand if the storage in question was managed. >> >>>>>>> >> >>>>>>> Then I could have created my SR/datastore in the CreateCommand and >> >>>>>>> the AttachCommand would have had the SR/datastore that it was >> always >> >>>>>>> expecting (and I wouldn't have had to create the SR/datastore in >> the >> >>>>>>> AttachCommand). >> >>>>>>> >> >>>>>>> >> >>>>>>> On Sat, Sep 21, 2013 at 11:56 PM, Marcus Sorensen < >> >>>>>>> shadow...@gmail.com> wrote: >> >>>>>>> >> >>>>>>>> Yeah, I think it probably is as well, but I figured you'd be in a >> >>>>>>>> better position to tell. >> >>>>>>>> >> >>>>>>>> I see that copyAsync is unsupported in your current 4.2 driver, >> does >> >>>>>>>> that mean that there's no template support? Or is it some other >> call >> >>>>>>>> that does templating now? I'm still getting up to speed on all >> of the >> >>>>>>>> 4.2 changes. I was just looking at CreateCommand in >> >>>>>>>> LibvirtComputingResource, since that's the only place >> >>>>>>>> createPhysicalDisk is called, and it occurred to me that >> >>>>>>>> CreateCommand >> >>>>>>>> might be skipped altogether when utilizing storage plugins. >> >>>>>>>> >> >>>>>>>> On Sat, Sep 21, 2013 at 11:38 PM, Mike Tutkowski >> >>>>>>>> <mike.tutkow...@solidfire.com> wrote: >> >>>>>>>> > That's an interesting comment, Marcus. >> >>>>>>>> > >> >>>>>>>> > It was my intent that it should work with any CloudStack >> "managed" >> >>>>>>>> storage >> >>>>>>>> > that uses an iSCSI target. Even though I'm using CHAP, I wrote >> the >> >>>>>>>> code so >> >>>>>>>> > CHAP didn't have to be used. >> >>>>>>>> > >> >>>>>>>> > As I'm doing my testing, I can try to think about whether it is >> >>>>>>>> generic >> >>>>>>>> > enough to keep those names or not. >> >>>>>>>> > >> >>>>>>>> > My expectation is that it is generic enough. >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > On Sat, Sep 21, 2013 at 11:32 PM, Marcus Sorensen < >> >>>>>>>> shadow...@gmail.com>wrote: >> >>>>>>>> > >> >>>>>>>> >> I added a comment to your diff. In general I think it looks >> good, >> >>>>>>>> >> though I obviously can't vouch for whether or not it will >> work. >> >>>>>>>> One >> >>>>>>>> >> thing I do have reservations about is the adaptor/pool >> naming. If >> >>>>>>>> you >> >>>>>>>> >> think the code is generic enough that it will work for anyone >> who >> >>>>>>>> does >> >>>>>>>> >> an iscsi LUN-per-volume plugin, then it's OK, but if there's >> >>>>>>>> anything >> >>>>>>>> >> about it that's specific to YOUR iscsi target or how it likes >> to >> >>>>>>>> be >> >>>>>>>> >> treated then I'd say that they should be named something less >> >>>>>>>> generic >> >>>>>>>> >> than iScsiAdmStorage. >> >>>>>>>> >> >> >>>>>>>> >> On Sat, Sep 21, 2013 at 7:23 PM, Mike Tutkowski >> >>>>>>>> >> <mike.tutkow...@solidfire.com> wrote: >> >>>>>>>> >> > Great - thanks! >> >>>>>>>> >> > >> >>>>>>>> >> > Just to give you an overview of what my code does (for when >> you >> >>>>>>>> get a >> >>>>>>>> >> > chance to review it): >> >>>>>>>> >> > >> >>>>>>>> >> > SolidFireHostListener is registered in >> >>>>>>>> SolidfirePrimaryDataStoreProvider. >> >>>>>>>> >> > Its hostConnect method is invoked when a host connects with >> the >> >>>>>>>> CS MS. If >> >>>>>>>> >> > the host is running KVM, the listener sends a >> >>>>>>>> ModifyStoragePoolCommand to >> >>>>>>>> >> > the host. This logic was based off of DefaultHostListener. >> >>>>>>>> >> > >> >>>>>>>> >> > The handling of ModifyStoragePoolCommand is unchanged. It >> >>>>>>>> invokes >> >>>>>>>> >> > createStoragePool on the KVMStoragePoolManager. The >> >>>>>>>> KVMStoragePoolManager >> >>>>>>>> >> > asks for an adaptor and finds my new one: >> >>>>>>>> iScsiAdmStorageAdaptor (which >> >>>>>>>> >> was >> >>>>>>>> >> > registered in the constructor for KVMStoragePoolManager >> under >> >>>>>>>> the key of >> >>>>>>>> >> > StoragePoolType.Iscsi.toString()). >> >>>>>>>> >> > >> >>>>>>>> >> > iScsiAdmStorageAdaptor.createStoragePool just makes an >> instance >> >>>>>>>> of >> >>>>>>>> >> > iScsiAdmStoragePool, adds it to a map, and returns the >> pointer >> >>>>>>>> to the >> >>>>>>>> >> > iScsiAdmStoragePool object. The key of the map is the UUID >> of >> >>>>>>>> the storage >> >>>>>>>> >> > pool. >> >>>>>>>> >> > >> >>>>>>>> >> > When a volume is attached, createPhysicalDisk is invoked for >> >>>>>>>> managed >> >>>>>>>> >> > storage rather than getPhysicalDisk. createPhysicalDisk uses >> >>>>>>>> iscsiadm to >> >>>>>>>> >> > establish the iSCSI connection to the volume on the SAN and >> a >> >>>>>>>> >> > KVMPhysicalDisk is returned to be used in the attach logic >> that >> >>>>>>>> follows. >> >>>>>>>> >> > >> >>>>>>>> >> > When a volume is detached, getPhysicalDisk is invoked with >> the >> >>>>>>>> IQN of the >> >>>>>>>> >> > volume if the storage pool in question is managed storage. >> >>>>>>>> Otherwise, the >> >>>>>>>> >> > normal vol.getPath() is used. >> >>>>>>>> iScsiAdmStorageAdaptor.getPhysicalDisk just >> >>>>>>>> >> > returns a new instance of KVMPhysicalDisk to be used in the >> >>>>>>>> detach logic. >> >>>>>>>> >> > >> >>>>>>>> >> > Once the volume has been detached, >> >>>>>>>> iScsiAdmStoragePool.deletePhysicalDisk >> >>>>>>>> >> > is invoked if the storage pool is managed. >> deletePhysicalDisk >> >>>>>>>> removes the >> >>>>>>>> >> > iSCSI connection to the volume using iscsiadm. >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> >> > On Sat, Sep 21, 2013 at 5:46 PM, Marcus Sorensen < >> >>>>>>>> shadow...@gmail.com >> >>>>>>>> >> >wrote: >> >>>>>>>> >> > >> >>>>>>>> >> >> Its the log4j properties file in /etc/cloudstack/agent >> change >> >>>>>>>> all INFO >> >>>>>>>> >> to >> >>>>>>>> >> >> DEBUG. I imagine the agent just isn't starting, you can >> tail >> >>>>>>>> the log >> >>>>>>>> >> when >> >>>>>>>> >> >> you try to start the service, or maybe it will spit >> something >> >>>>>>>> out into >> >>>>>>>> >> one >> >>>>>>>> >> >> of the other files in /var/log/cloudstack/agent >> >>>>>>>> >> >> On Sep 21, 2013 5:19 PM, "Mike Tutkowski" < >> >>>>>>>> mike.tutkow...@solidfire.com >> >>>>>>>> >> > >> >>>>>>>> >> >> wrote: >> >>>>>>>> >> >> >> >>>>>>>> >> >> > This is how I've been trying to query for the status of >> the >> >>>>>>>> service (I >> >>>>>>>> >> >> > assume it could be started this way, as well, by changing >> >>>>>>>> "status" to >> >>>>>>>> >> >> > "start" or "restart"?): >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo >> >>>>>>>> /usr/sbin/service >> >>>>>>>> >> >> > cloudstack-agent status >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > I get this back: >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > Failed to execute: * could not access PID file for >> >>>>>>>> cloudstack-agent >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > I've made a bunch of code changes recently, though, so I >> >>>>>>>> think I'm >> >>>>>>>> >> going >> >>>>>>>> >> >> to >> >>>>>>>> >> >> > rebuild and redeploy everything. >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > The debug info sounds helpful. Where can I set >> enable.debug? >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > Thanks, Marcus! >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > On Sat, Sep 21, 2013 at 4:11 PM, Marcus Sorensen < >> >>>>>>>> shadow...@gmail.com >> >>>>>>>> >> >> > >wrote: >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > > OK, will check it out in the next few days. As >> mentioned, >> >>>>>>>> you can >> >>>>>>>> >> set >> >>>>>>>> >> >> up >> >>>>>>>> >> >> > > your Ubuntu vm as the management server as well if all >> >>>>>>>> else fails. >> >>>>>>>> >> If >> >>>>>>>> >> >> > you >> >>>>>>>> >> >> > > can get to the mgmt server on 8250 from the KVM host, >> then >> >>>>>>>> you need >> >>>>>>>> >> to >> >>>>>>>> >> >> > > enable.debug on the agent. It won't run without >> >>>>>>>> complaining loudly >> >>>>>>>> >> if >> >>>>>>>> >> >> it >> >>>>>>>> >> >> > > can't get to the mgmt server, and I didn't see that in >> >>>>>>>> your agent >> >>>>>>>> >> log, >> >>>>>>>> >> >> so >> >>>>>>>> >> >> > > perhaps its not running. I assume you know how to >> >>>>>>>> stop/start the >> >>>>>>>> >> agent >> >>>>>>>> >> >> on >> >>>>>>>> >> >> > > KVM via 'service cloud stacks agent'. >> >>>>>>>> >> >> > > On Sep 21, 2013 3:02 PM, "Mike Tutkowski" < >> >>>>>>>> >> >> mike.tutkow...@solidfire.com> >> >>>>>>>> >> >> > > wrote: >> >>>>>>>> >> >> > > >> >>>>>>>> >> >> > > > Hey Marcus, >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > I haven't yet been able to test my new code, but I >> >>>>>>>> thought you >> >>>>>>>> >> would >> >>>>>>>> >> >> > be a >> >>>>>>>> >> >> > > > good person to ask to review it: >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > >> >>>>>>>> >> >> > >> >>>>>>>> >> >> >> >>>>>>>> >> >> >>>>>>>> >> https://github.com/mike-tutkowski/incubator-cloudstack/commit/ea74b312a8a36801994500407fd54f0cdda55e37 >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > All it is supposed to do is attach and detach a data >> >>>>>>>> disk (that >> >>>>>>>> >> has >> >>>>>>>> >> >> > > > guaranteed IOPS) with KVM as the hypervisor. The data >> >>>>>>>> disk >> >>>>>>>> >> happens to >> >>>>>>>> >> >> > be >> >>>>>>>> >> >> > > > from SolidFire-backed storage - where we have a 1:1 >> >>>>>>>> mapping >> >>>>>>>> >> between a >> >>>>>>>> >> >> > > > CloudStack volume and a data disk. >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > There is no support for hypervisor snapshots or stuff >> >>>>>>>> like that >> >>>>>>>> >> >> > (likely a >> >>>>>>>> >> >> > > > future release)...just attaching and detaching a data >> >>>>>>>> disk in 4.3. >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > Thanks! >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > On Fri, Sep 20, 2013 at 11:05 PM, Mike Tutkowski < >> >>>>>>>> >> >> > > > mike.tutkow...@solidfire.com> wrote: >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > > When I re-deployed the DEBs, I didn't remove >> >>>>>>>> cloudstack-agent >> >>>>>>>> >> >> first. >> >>>>>>>> >> >> > > > Would >> >>>>>>>> >> >> > > > > that be a problem? I just did a sudo apt-get >> install >> >>>>>>>> >> >> > cloudstack-agent. >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > > On Fri, Sep 20, 2013 at 11:03 PM, Mike Tutkowski < >> >>>>>>>> >> >> > > > > mike.tutkow...@solidfire.com> wrote: >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > >> I get the same error running the command manually: >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >> mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo >> >>>>>>>> >> /usr/sbin/service >> >>>>>>>> >> >> > > > >> cloudstack-agent status >> >>>>>>>> >> >> > > > >> * could not access PID file for cloudstack-agent >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >> On Fri, Sep 20, 2013 at 11:00 PM, Mike Tutkowski < >> >>>>>>>> >> >> > > > >> mike.tutkow...@solidfire.com> wrote: >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >>> agent.log looks OK to me: >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,010 INFO >> >>>>>>>> [cloud.agent.AgentShell] >> >>>>>>>> >> >> (main:null) >> >>>>>>>> >> >> > > > Agent >> >>>>>>>> >> >> > > > >>> started >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,011 INFO >> >>>>>>>> [cloud.agent.AgentShell] >> >>>>>>>> >> >> (main:null) >> >>>>>>>> >> >> > > > >>> Implementation Version is 4.3.0-SNAPSHOT >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,015 INFO >> >>>>>>>> [cloud.agent.AgentShell] >> >>>>>>>> >> >> (main:null) >> >>>>>>>> >> >> > > > >>> agent.properties found at >> >>>>>>>> >> /etc/cloudstack/agent/agent.properties >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,023 INFO >> >>>>>>>> [cloud.agent.AgentShell] >> >>>>>>>> >> >> (main:null) >> >>>>>>>> >> >> > > > >>> Defaulting to using properties file for storage >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,024 INFO >> >>>>>>>> [cloud.agent.AgentShell] >> >>>>>>>> >> >> (main:null) >> >>>>>>>> >> >> > > > >>> Defaulting to the constant time backoff algorithm >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,029 INFO >> [cloud.utils.LogUtils] >> >>>>>>>> >> (main:null) >> >>>>>>>> >> >> > > log4j >> >>>>>>>> >> >> > > > >>> configuration found at >> >>>>>>>> /etc/cloudstack/agent/log4j-cloud.xml >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,163 INFO [cloud.agent.Agent] >> >>>>>>>> (main:null) >> >>>>>>>> >> id >> >>>>>>>> >> >> > is 3 >> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,197 INFO >> >>>>>>>> >> >> > > > >>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>>>> (main:null) >> >>>>>>>> >> >> > > > >>> VirtualRoutingResource _scriptDir to use: >> >>>>>>>> >> >> scripts/network/domr/kvm >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> However, I wasn't aware that setup.log was >> >>>>>>>> important. This >> >>>>>>>> >> seems >> >>>>>>>> >> >> to >> >>>>>>>> >> >> > > be >> >>>>>>>> >> >> > > > a >> >>>>>>>> >> >> > > > >>> problem, but I'm not sure what it might indicate: >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service >> >>>>>>>> cloudstack-agent >> >>>>>>>> >> status >> >>>>>>>> >> >> > > > >>> DEBUG:root:Failed to execute: * could not access >> PID >> >>>>>>>> file for >> >>>>>>>> >> >> > > > >>> cloudstack-agent >> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service >> >>>>>>>> cloudstack-agent >> >>>>>>>> >> start >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> On Fri, Sep 20, 2013 at 10:53 PM, Marcus >> Sorensen < >> >>>>>>>> >> >> > > shadow...@gmail.com >> >>>>>>>> >> >> > > > >wrote: >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>>> Sorry, I saw that in the log, I thought it was >> the >> >>>>>>>> agent log >> >>>>>>>> >> for >> >>>>>>>> >> >> > > some >> >>>>>>>> >> >> > > > >>>> reason. Is the agent started? That might be the >> >>>>>>>> place to >> >>>>>>>> >> look. >> >>>>>>>> >> >> > There >> >>>>>>>> >> >> > > > is >> >>>>>>>> >> >> > > > >>>> an >> >>>>>>>> >> >> > > > >>>> agent log for the agent and one for the setup >> when >> >>>>>>>> it adds >> >>>>>>>> >> the >> >>>>>>>> >> >> > host, >> >>>>>>>> >> >> > > > >>>> both >> >>>>>>>> >> >> > > > >>>> in /var/log >> >>>>>>>> >> >> > > > >>>> On Sep 20, 2013 10:42 PM, "Mike Tutkowski" < >> >>>>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com> >> >>>>>>>> >> >> > > > >>>> wrote: >> >>>>>>>> >> >> > > > >>>> >> >>>>>>>> >> >> > > > >>>> > Is it saying that the MS is at the IP address >> or >> >>>>>>>> the KVM >> >>>>>>>> >> host? >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > The KVM host is at 192.168.233.10. >> >>>>>>>> >> >> > > > >>>> > The MS host is at 192.168.233.1. >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > I see this for my host Global Settings >> parameter: >> >>>>>>>> >> >> > > > >>>> > hostThe ip address of management >> >>>>>>>> server192.168.233.1 >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > /etc/cloudstack/agent/agent.properties has a >> >>>>>>>> >> >> host=192.168.233.1 >> >>>>>>>> >> >> > > > value. >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > On Fri, Sep 20, 2013 at 10:21 PM, Marcus >> Sorensen >> >>>>>>>> < >> >>>>>>>> >> >> > > > >>>> shadow...@gmail.com >> >>>>>>>> >> >> > > > >>>> > >wrote: >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > > The log says your mgmt server is >> >>>>>>>> 192.168.233.10? But you >> >>>>>>>> >> >> tried >> >>>>>>>> >> >> > > to >> >>>>>>>> >> >> > > > >>>> telnet >> >>>>>>>> >> >> > > > >>>> > to >> >>>>>>>> >> >> > > > >>>> > > 192.168.233.1? It might be enough to change >> >>>>>>>> that in >> >>>>>>>> >> >> > > > >>>> > > /etc/cloudstack/agent/agent.properties, but >> you >> >>>>>>>> may want >> >>>>>>>> >> to >> >>>>>>>> >> >> > edit >> >>>>>>>> >> >> > > > the >> >>>>>>>> >> >> > > > >>>> > config >> >>>>>>>> >> >> > > > >>>> > > as well to tell it the real ms IP. >> >>>>>>>> >> >> > > > >>>> > > On Sep 20, 2013 10:12 PM, "Mike Tutkowski" < >> >>>>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > >>>> > > >> >>>>>>>> >> >> > > > >>>> > > wrote: >> >>>>>>>> >> >> > > > >>>> > > >> >>>>>>>> >> >> > > > >>>> > > > Here's what my /etc/network/interfaces >> file >> >>>>>>>> looks >> >>>>>>>> >> like, if >> >>>>>>>> >> >> > > that >> >>>>>>>> >> >> > > > >>>> is of >> >>>>>>>> >> >> > > > >>>> > > > interest (the 192.168.233.0 network is the >> >>>>>>>> NAT network >> >>>>>>>> >> >> > VMware >> >>>>>>>> >> >> > > > >>>> Fusion >> >>>>>>>> >> >> > > > >>>> > set >> >>>>>>>> >> >> > > > >>>> > > > up): >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > auto lo >> >>>>>>>> >> >> > > > >>>> > > > iface lo inet loopback >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > auto eth0 >> >>>>>>>> >> >> > > > >>>> > > > iface eth0 inet manual >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > auto cloudbr0 >> >>>>>>>> >> >> > > > >>>> > > > iface cloudbr0 inet static >> >>>>>>>> >> >> > > > >>>> > > > address 192.168.233.10 >> >>>>>>>> >> >> > > > >>>> > > > netmask 255.255.255.0 >> >>>>>>>> >> >> > > > >>>> > > > network 192.168.233.0 >> >>>>>>>> >> >> > > > >>>> > > > broadcast 192.168.233.255 >> >>>>>>>> >> >> > > > >>>> > > > dns-nameservers 8.8.8.8 >> >>>>>>>> >> >> > > > >>>> > > > bridge_ports eth0 >> >>>>>>>> >> >> > > > >>>> > > > bridge_fd 5 >> >>>>>>>> >> >> > > > >>>> > > > bridge_stp off >> >>>>>>>> >> >> > > > >>>> > > > bridge_maxwait 1 >> >>>>>>>> >> >> > > > >>>> > > > post-up route add default gw >> >>>>>>>> 192.168.233.2 metric 1 >> >>>>>>>> >> >> > > > >>>> > > > pre-down route del default gw >> >>>>>>>> 192.168.233.2 >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > On Fri, Sep 20, 2013 at 10:08 PM, Mike >> >>>>>>>> Tutkowski < >> >>>>>>>> >> >> > > > >>>> > > > mike.tutkow...@solidfire.com> wrote: >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > > You appear to be correct. This is from >> the >> >>>>>>>> MS log >> >>>>>>>> >> >> (below). >> >>>>>>>> >> >> > > > >>>> Discovery >> >>>>>>>> >> >> > > > >>>> > > > timed >> >>>>>>>> >> >> > > > >>>> > > > > out. >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > I'm not sure why this would be. My >> network >> >>>>>>>> settings >> >>>>>>>> >> >> > > shouldn't >> >>>>>>>> >> >> > > > >>>> have >> >>>>>>>> >> >> > > > >>>> > > > changed >> >>>>>>>> >> >> > > > >>>> > > > > since the last time I tried this. >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > I am able to ping the KVM host from the >> MS >> >>>>>>>> host and >> >>>>>>>> >> vice >> >>>>>>>> >> >> > > > versa. >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > I'm even able to manually kick off a VM >> on >> >>>>>>>> the KVM >> >>>>>>>> >> host >> >>>>>>>> >> >> > and >> >>>>>>>> >> >> > > > >>>> ping from >> >>>>>>>> >> >> > > > >>>> > > it >> >>>>>>>> >> >> > > > >>>> > > > > to the MS host. >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > I am using NAT from my Mac OS X host >> (also >> >>>>>>>> running >> >>>>>>>> >> the >> >>>>>>>> >> >> CS >> >>>>>>>> >> >> > > MS) >> >>>>>>>> >> >> > > > >>>> to the >> >>>>>>>> >> >> > > > >>>> > VM >> >>>>>>>> >> >> > > > >>>> > > > > running KVM (VMware Fusion). >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,141 DEBUG >> >>>>>>>> >> >> > > > >>>> [c.c.h.k.d.LibvirtServerDiscoverer] >> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >> :ctx-6b28dc48) >> >>>>>>>> Timeout, >> >>>>>>>> >> to >> >>>>>>>> >> >> > wait >> >>>>>>>> >> >> > > > for >> >>>>>>>> >> >> > > > >>>> the >> >>>>>>>> >> >> > > > >>>> > > host >> >>>>>>>> >> >> > > > >>>> > > > > connecting to mgt svr, assuming it is >> failed >> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,144 WARN >> >>>>>>>> >> >> [c.c.r.ResourceManagerImpl] >> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >> :ctx-6b28dc48) >> >>>>>>>> Unable to >> >>>>>>>> >> >> find >> >>>>>>>> >> >> > > the >> >>>>>>>> >> >> > > > >>>> server >> >>>>>>>> >> >> > > > >>>> > > > > resources at http://192.168.233.10 >> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,145 INFO >> >>>>>>>> >> >> > [c.c.u.e.CSExceptionErrorCode] >> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >> :ctx-6b28dc48) >> >>>>>>>> Could not >> >>>>>>>> >> >> find >> >>>>>>>> >> >> > > > >>>> exception: >> >>>>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException >> in >> >>>>>>>> error code >> >>>>>>>> >> >> list >> >>>>>>>> >> >> > > for >> >>>>>>>> >> >> > > > >>>> > > exceptions >> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,147 WARN >> >>>>>>>> >> [o.a.c.a.c.a.h.AddHostCmd] >> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >> :ctx-6b28dc48) >> >>>>>>>> Exception: >> >>>>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException: >> >>>>>>>> Unable to add >> >>>>>>>> >> >> the >> >>>>>>>> >> >> > > host >> >>>>>>>> >> >> > > > >>>> > > > > at >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > >> >>>>>>>> >> >> > >> >>>>>>>> >> >> >> >>>>>>>> >> >> >>>>>>>> >> com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:778) >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > I do seem to be able to telnet in from >> my >> >>>>>>>> KVM host to >> >>>>>>>> >> >> the >> >>>>>>>> >> >> > MS >> >>>>>>>> >> >> > > > >>>> host's >> >>>>>>>> >> >> > > > >>>> > > 8250 >> >>>>>>>> >> >> > > > >>>> > > > > port: >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > > mtutkowski@ubuntu:~$ telnet >> 192.168.233.1 >> >>>>>>>> 8250 >> >>>>>>>> >> >> > > > >>>> > > > > Trying 192.168.233.1... >> >>>>>>>> >> >> > > > >>>> > > > > Connected to 192.168.233.1. >> >>>>>>>> >> >> > > > >>>> > > > > Escape character is '^]'. >> >>>>>>>> >> >> > > > >>>> > > > > >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > > -- >> >>>>>>>> >> >> > > > >>>> > > > *Mike Tutkowski* >> >>>>>>>> >> >> > > > >>>> > > > *Senior CloudStack Developer, SolidFire >> Inc.* >> >>>>>>>> >> >> > > > >>>> > > > e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > >>>> > > > o: 303.746.7302 >> >>>>>>>> >> >> > > > >>>> > > > Advancing the way the world uses the >> >>>>>>>> >> >> > > > >>>> > > > cloud< >> >>>>>>>> >> http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> >> >> > > > >>>> > > > *™* >> >>>>>>>> >> >> > > > >>>> > > > >> >>>>>>>> >> >> > > > >>>> > > >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> > -- >> >>>>>>>> >> >> > > > >>>> > *Mike Tutkowski* >> >>>>>>>> >> >> > > > >>>> > *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> >> > > > >>>> > e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > >>>> > o: 303.746.7302 >> >>>>>>>> >> >> > > > >>>> > Advancing the way the world uses the >> >>>>>>>> >> >> > > > >>>> > cloud< >> >>>>>>>> http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> >> >> > > > >>>> > *™* >> >>>>>>>> >> >> > > > >>>> > >> >>>>>>>> >> >> > > > >>>> >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >>> -- >> >>>>>>>> >> >> > > > >>> *Mike Tutkowski* >> >>>>>>>> >> >> > > > >>> *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> >> > > > >>> e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > >>> o: 303.746.7302 >> >>>>>>>> >> >> > > > >>> Advancing the way the world uses the cloud< >> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> >> >> > > > >>> *™* >> >>>>>>>> >> >> > > > >>> >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > >> -- >> >>>>>>>> >> >> > > > >> *Mike Tutkowski* >> >>>>>>>> >> >> > > > >> *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> >> > > > >> e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > >> o: 303.746.7302 >> >>>>>>>> >> >> > > > >> Advancing the way the world uses the cloud< >> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> >> >> > > > >> *™* >> >>>>>>>> >> >> > > > >> >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > > -- >> >>>>>>>> >> >> > > > > *Mike Tutkowski* >> >>>>>>>> >> >> > > > > *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> >> > > > > e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > > o: 303.746.7302 >> >>>>>>>> >> >> > > > > Advancing the way the world uses the cloud< >> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> >> >> > > > > *™* >> >>>>>>>> >> >> > > > > >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > > -- >> >>>>>>>> >> >> > > > *Mike Tutkowski* >> >>>>>>>> >> >> > > > *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> >> > > > e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > > > o: 303.746.7302 >> >>>>>>>> >> >> > > > Advancing the way the world uses the >> >>>>>>>> >> >> > > > cloud< >> http://solidfire.com/solution/overview/?video=play >> >>>>>>>> > >> >>>>>>>> >> >> > > > *™* >> >>>>>>>> >> >> > > > >> >>>>>>>> >> >> > > >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > >> >>>>>>>> >> >> > -- >> >>>>>>>> >> >> > *Mike Tutkowski* >> >>>>>>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> >> > e: mike.tutkow...@solidfire.com >> >>>>>>>> >> >> > o: 303.746.7302 >> >>>>>>>> >> >> > Advancing the way the world uses the >> >>>>>>>> >> >> > cloud<http://solidfire.com/solution/overview/?video=play >> > >> >>>>>>>> >> >> > *™* >> >>>>>>>> >> >> > >> >>>>>>>> >> >> >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> >> > -- >> >>>>>>>> >> > *Mike Tutkowski* >> >>>>>>>> >> > *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> >> > e: mike.tutkow...@solidfire.com >> >>>>>>>> >> > o: 303.746.7302 >> >>>>>>>> >> > Advancing the way the world uses the >> >>>>>>>> >> > cloud<http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> >> > *™* >> >>>>>>>> >> >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > -- >> >>>>>>>> > *Mike Tutkowski* >> >>>>>>>> > *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>>> > e: mike.tutkow...@solidfire.com >> >>>>>>>> > o: 303.746.7302 >> >>>>>>>> > Advancing the way the world uses the >> >>>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play> >> >>>>>>>> > *™* >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> *Mike Tutkowski* >> >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>>> e: mike.tutkow...@solidfire.com >> >>>>>>> o: 303.746.7302 >> >>>>>>> Advancing the way the world uses the cloud< >> http://solidfire.com/solution/overview/?video=play> >> >>>>>>> *™* >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> *Mike Tutkowski* >> >>>>>> *Senior CloudStack Developer, SolidFire Inc.* >> >>>>>> e: mike.tutkow...@solidfire.com >> >>>>>> o: 303.746.7302 >> >>>>>> Advancing the way the world uses the cloud< >> http://solidfire.com/solution/overview/?video=play> >> >>>>>> *™* >> >>>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> *Mike Tutkowski* >> >>>>> *Senior CloudStack Developer, SolidFire Inc.* >> >>>>> e: mike.tutkow...@solidfire.com >> >>>>> o: 303.746.7302 >> >>>>> Advancing the way the world uses the cloud< >> http://solidfire.com/solution/overview/?video=play> >> >>>>> *™* >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> *Mike Tutkowski* >> >>>> *Senior CloudStack Developer, SolidFire Inc.* >> >>>> e: mike.tutkow...@solidfire.com >> >>>> o: 303.746.7302 >> >>>> Advancing the way the world uses the cloud< >> http://solidfire.com/solution/overview/?video=play> >> >>>> *™* >> >>>> >> >>> >> >>> >> >>> >> >>> -- >> >>> *Mike Tutkowski* >> >>> *Senior CloudStack Developer, SolidFire Inc.* >> >>> e: mike.tutkow...@solidfire.com >> >>> o: 303.746.7302 >> >>> Advancing the way the world uses the cloud< >> http://solidfire.com/solution/overview/?video=play> >> >>> *™* >> >>> >> >> >> >> >> >> >> >> -- >> >> *Mike Tutkowski* >> >> *Senior CloudStack Developer, SolidFire Inc.* >> >> e: mike.tutkow...@solidfire.com >> >> o: 303.746.7302 >> >> Advancing the way the world uses the cloud< >> http://solidfire.com/solution/overview/?video=play> >> >> *™* >> >> >> > >> > >> > >> > -- >> > *Mike Tutkowski* >> > *Senior CloudStack Developer, SolidFire Inc.* >> > e: mike.tutkow...@solidfire.com >> > o: 303.746.7302 >> > Advancing the way the world uses the >> > cloud<http://solidfire.com/solution/overview/?video=play> >> > *™* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
