maybe cloudstack-agent.out
On Mon, Sep 23, 2013 at 4:44 PM, Mike Tutkowski <mike.tutkow...@solidfire.com> wrote: > OK, so, nothing is screaming out in the logs. I did notice the following: > > From setup.log: > > DEBUG:root:execute:apparmor_status |grep libvirt > > DEBUG:root:Failed to execute: > > > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status > > DEBUG:root:Failed to execute: * could not access PID file for > cloudstack-agent > > > This is the final line in this log file: > > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start > > > This is from agent.log: > > 2013-09-23 15:30:55,549 DEBUG [cloud.agent.AgentShell] (main:null) Checking > to see if agent.pid exists. > > 2013-09-23 15:30:55,655 DEBUG [cloud.utils.ProcessUtil] (main:null) > Executing: bash -c echo $PPID > > 2013-09-23 15:30:55,742 DEBUG [cloud.utils.ProcessUtil] (main:null) > Execution is successful. > > 2013-09-23 15:30:56,000 INFO [cloud.agent.Agent] (main:null) id is > > 2013-09-23 15:30:56,000 DEBUG [cloud.resource.ServerResourceBase] > (main:null) Retrieving network interface: cloudbr0 > > 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase] > (main:null) Retrieving network interface: cloudbr0 > > 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase] > (main:null) Retrieving network interface: null > > 2013-09-23 15:30:56,017 DEBUG [cloud.resource.ServerResourceBase] > (main:null) Retrieving network interface: null > > > The following kinds of lines are repeated for a bunch of different .sh > files. I think they often end up being found here: > /usr/share/cloudstack-common/scripts/network/domr, so this is probably not > an issue. > > > 2013-09-23 15:30:56,111 DEBUG [utils.script.Script] (main:null) Looking for > call_firewall.sh in the classpath > > 2013-09-23 15:30:56,112 DEBUG [utils.script.Script] (main:null) System > resource: null > > 2013-09-23 15:30:56,113 DEBUG [utils.script.Script] (main:null) Classpath > resource: null > > 2013-09-23 15:30:56,123 DEBUG [utils.script.Script] (main:null) Looking for > call_firewall.sh > > > Is there a log file for the Java code that I could write stuff out to and > see how far we get? > > > On Mon, Sep 23, 2013 at 3:17 PM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > >> Thanks, Marcus >> >> I've been developing on Windows for most of my time, so a bunch of these >> Linux-type commands are new to me and I don't always interpret the output >> correctly. Getting there. :) >> >> >> On Mon, Sep 23, 2013 at 2:37 PM, Marcus Sorensen <shadow...@gmail.com>wrote: >> >>> Nope, not running. That's just your grep process. It would look like: >>> >>> root 24429 24428 1 14:25 ? 00:00:08 jsvc.exec -cp >>> >>> /usr/share/java/commons-daemon.jar:/usr/share/java/jna.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjrt-1.7. >>> >>> Your agent log should tell you why it failed to start if you set it in >>> debug and try to start... or maybe cloudstack-agent.out if it doesn't >>> get far enough (say it's missing a class or something and can't >>> start). >>> >>> On Mon, Sep 23, 2013 at 2:33 PM, Mike Tutkowski >>> <mike.tutkow...@solidfire.com> wrote: >>> > Looks like it's running, though: >>> > >>> > mtutkowski@ubuntu:~$ ps -ef | grep jsvc >>> > 1000 7097 7013 0 14:32 pts/1 00:00:00 grep --color=auto jsvc >>> > >>> > >>> > >>> > On Mon, Sep 23, 2013 at 2:31 PM, Mike Tutkowski < >>> > mike.tutkow...@solidfire.com> wrote: >>> > >>> >> Hey Marcus, >>> >> >>> >> Maybe you could give me a better idea of what the "flow" is when >>> adding a >>> >> KVM host. >>> >> >>> >> It looks like we SSH into the potential KVM host and execute a startup >>> >> script (giving it necessary info about the cloud and the management >>> server >>> >> it should talk to). >>> >> >>> >> After this, is the Java VM started? >>> >> >>> >> After a reboot, I assume the JVM is started automatically? >>> >> >>> >> How do you debug your KVM-side Java code? >>> >> >>> >> Been looking through the logs and nothing obvious sticks out. I will >>> have >>> >> another look. >>> >> >>> >> Thanks >>> >> >>> >> >>> >> On Mon, Sep 23, 2013 at 2:15 PM, Mike Tutkowski < >>> >> mike.tutkow...@solidfire.com> wrote: >>> >> >>> >>> Hey Marcus, >>> >>> >>> >>> I've been investigating my issue with not being able to add a KVM >>> host to >>> >>> CS. >>> >>> >>> >>> For what it's worth, this comes back successful: >>> >>> >>> >>> SSHCmdHelper.sshExecuteCmd(sshConnection, "cloudstack-setup-agent " + >>> >>> parameters, 3); >>> >>> >>> >>> This is what the command looks like: >>> >>> >>> >>> cloudstack-setup-agent -m 192.168.233.1 -z 1 -p 1 -c 1 -g >>> >>> 6b4aa1c2-2ac9-3c60-aabe-704aed40c684 -a --pubNic=cloudbr0 >>> --prvNic=cloudbr0 >>> >>> --guestNic=cloudbr0 >>> >>> >>> >>> The problem is this method in LibvirtServerDiscoverer never finds a >>> >>> matching host in the DB: >>> >>> >>> >>> waitForHostConnect(long dcId, long podId, long clusterId, String guid) >>> >>> >>> >>> I assume once the KVM host is up and running that it's supposed to >>> call >>> >>> into the CS MS so the DB can be updated as such? >>> >>> >>> >>> If so, the problem must be on the KVM side. >>> >>> >>> >>> I did run this again (from the KVM host) to see if the connection was >>> in >>> >>> place: >>> >>> >>> >>> mtutkowski@ubuntu:~$ telnet 192.168.233.1 8250 >>> >>> >>> >>> Trying 192.168.233.1... >>> >>> >>> >>> Connected to 192.168.233.1. >>> >>> >>> >>> Escape character is '^]'. >>> >>> So that looks good. >>> >>> >>> >>> I turned on more info in the debug log, but nothing obvious jumps out >>> as >>> >>> of yet. >>> >>> >>> >>> If you have any thoughts on this, please shoot them my way. :) >>> >>> >>> >>> Thanks! >>> >>> >>> >>> >>> >>> On Sun, Sep 22, 2013 at 12:11 AM, Mike Tutkowski < >>> >>> mike.tutkow...@solidfire.com> wrote: >>> >>> >>> >>>> First step is for me to get this working for KVM, though. :) >>> >>>> >>> >>>> Once I do that, I can perhaps make modifications to the storage >>> >>>> framework and hypervisor plug-ins to refactor the logic and such. >>> >>>> >>> >>>> >>> >>>> On Sun, Sep 22, 2013 at 12:09 AM, Mike Tutkowski < >>> >>>> mike.tutkow...@solidfire.com> wrote: >>> >>>> >>> >>>>> Same would work for KVM. >>> >>>>> >>> >>>>> If CreateCommand and DestroyCommand were called at the appropriate >>> >>>>> times by the storage framework, I could move my connect and >>> disconnect >>> >>>>> logic out of the attach/detach logic. >>> >>>>> >>> >>>>> >>> >>>>> On Sun, Sep 22, 2013 at 12:08 AM, Mike Tutkowski < >>> >>>>> mike.tutkow...@solidfire.com> wrote: >>> >>>>> >>> >>>>>> Conversely, if the storage framework called the DestroyCommand for >>> >>>>>> managed storage after the DetachCommand, then I could have had my >>> remove >>> >>>>>> SR/datastore logic placed in the DestroyCommand handling rather >>> than in the >>> >>>>>> DetachCommand handling. >>> >>>>>> >>> >>>>>> >>> >>>>>> On Sun, Sep 22, 2013 at 12:06 AM, Mike Tutkowski < >>> >>>>>> mike.tutkow...@solidfire.com> wrote: >>> >>>>>> >>> >>>>>>> Edison's plug-in calls the CreateCommand. Mine does not. >>> >>>>>>> >>> >>>>>>> The initial approach that was discussed during 4.2 was for me to >>> >>>>>>> modify the attach/detach logic only in the XenServer and VMware >>> hypervisor >>> >>>>>>> plug-ins. >>> >>>>>>> >>> >>>>>>> Now that I think about it more, though, I kind of would have >>> liked to >>> >>>>>>> have the storage framework send a CreateCommand to the hypervisor >>> before >>> >>>>>>> sending the AttachCommand if the storage in question was managed. >>> >>>>>>> >>> >>>>>>> Then I could have created my SR/datastore in the CreateCommand and >>> >>>>>>> the AttachCommand would have had the SR/datastore that it was >>> always >>> >>>>>>> expecting (and I wouldn't have had to create the SR/datastore in >>> the >>> >>>>>>> AttachCommand). >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> On Sat, Sep 21, 2013 at 11:56 PM, Marcus Sorensen < >>> >>>>>>> shadow...@gmail.com> wrote: >>> >>>>>>> >>> >>>>>>>> Yeah, I think it probably is as well, but I figured you'd be in a >>> >>>>>>>> better position to tell. >>> >>>>>>>> >>> >>>>>>>> I see that copyAsync is unsupported in your current 4.2 driver, >>> does >>> >>>>>>>> that mean that there's no template support? Or is it some other >>> call >>> >>>>>>>> that does templating now? I'm still getting up to speed on all >>> of the >>> >>>>>>>> 4.2 changes. I was just looking at CreateCommand in >>> >>>>>>>> LibvirtComputingResource, since that's the only place >>> >>>>>>>> createPhysicalDisk is called, and it occurred to me that >>> >>>>>>>> CreateCommand >>> >>>>>>>> might be skipped altogether when utilizing storage plugins. >>> >>>>>>>> >>> >>>>>>>> On Sat, Sep 21, 2013 at 11:38 PM, Mike Tutkowski >>> >>>>>>>> <mike.tutkow...@solidfire.com> wrote: >>> >>>>>>>> > That's an interesting comment, Marcus. >>> >>>>>>>> > >>> >>>>>>>> > It was my intent that it should work with any CloudStack >>> "managed" >>> >>>>>>>> storage >>> >>>>>>>> > that uses an iSCSI target. Even though I'm using CHAP, I wrote >>> the >>> >>>>>>>> code so >>> >>>>>>>> > CHAP didn't have to be used. >>> >>>>>>>> > >>> >>>>>>>> > As I'm doing my testing, I can try to think about whether it is >>> >>>>>>>> generic >>> >>>>>>>> > enough to keep those names or not. >>> >>>>>>>> > >>> >>>>>>>> > My expectation is that it is generic enough. >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > On Sat, Sep 21, 2013 at 11:32 PM, Marcus Sorensen < >>> >>>>>>>> shadow...@gmail.com>wrote: >>> >>>>>>>> > >>> >>>>>>>> >> I added a comment to your diff. In general I think it looks >>> good, >>> >>>>>>>> >> though I obviously can't vouch for whether or not it will >>> work. >>> >>>>>>>> One >>> >>>>>>>> >> thing I do have reservations about is the adaptor/pool >>> naming. If >>> >>>>>>>> you >>> >>>>>>>> >> think the code is generic enough that it will work for anyone >>> who >>> >>>>>>>> does >>> >>>>>>>> >> an iscsi LUN-per-volume plugin, then it's OK, but if there's >>> >>>>>>>> anything >>> >>>>>>>> >> about it that's specific to YOUR iscsi target or how it likes >>> to >>> >>>>>>>> be >>> >>>>>>>> >> treated then I'd say that they should be named something less >>> >>>>>>>> generic >>> >>>>>>>> >> than iScsiAdmStorage. >>> >>>>>>>> >> >>> >>>>>>>> >> On Sat, Sep 21, 2013 at 7:23 PM, Mike Tutkowski >>> >>>>>>>> >> <mike.tutkow...@solidfire.com> wrote: >>> >>>>>>>> >> > Great - thanks! >>> >>>>>>>> >> > >>> >>>>>>>> >> > Just to give you an overview of what my code does (for when >>> you >>> >>>>>>>> get a >>> >>>>>>>> >> > chance to review it): >>> >>>>>>>> >> > >>> >>>>>>>> >> > SolidFireHostListener is registered in >>> >>>>>>>> SolidfirePrimaryDataStoreProvider. >>> >>>>>>>> >> > Its hostConnect method is invoked when a host connects with >>> the >>> >>>>>>>> CS MS. If >>> >>>>>>>> >> > the host is running KVM, the listener sends a >>> >>>>>>>> ModifyStoragePoolCommand to >>> >>>>>>>> >> > the host. This logic was based off of DefaultHostListener. >>> >>>>>>>> >> > >>> >>>>>>>> >> > The handling of ModifyStoragePoolCommand is unchanged. It >>> >>>>>>>> invokes >>> >>>>>>>> >> > createStoragePool on the KVMStoragePoolManager. The >>> >>>>>>>> KVMStoragePoolManager >>> >>>>>>>> >> > asks for an adaptor and finds my new one: >>> >>>>>>>> iScsiAdmStorageAdaptor (which >>> >>>>>>>> >> was >>> >>>>>>>> >> > registered in the constructor for KVMStoragePoolManager >>> under >>> >>>>>>>> the key of >>> >>>>>>>> >> > StoragePoolType.Iscsi.toString()). >>> >>>>>>>> >> > >>> >>>>>>>> >> > iScsiAdmStorageAdaptor.createStoragePool just makes an >>> instance >>> >>>>>>>> of >>> >>>>>>>> >> > iScsiAdmStoragePool, adds it to a map, and returns the >>> pointer >>> >>>>>>>> to the >>> >>>>>>>> >> > iScsiAdmStoragePool object. The key of the map is the UUID >>> of >>> >>>>>>>> the storage >>> >>>>>>>> >> > pool. >>> >>>>>>>> >> > >>> >>>>>>>> >> > When a volume is attached, createPhysicalDisk is invoked for >>> >>>>>>>> managed >>> >>>>>>>> >> > storage rather than getPhysicalDisk. createPhysicalDisk uses >>> >>>>>>>> iscsiadm to >>> >>>>>>>> >> > establish the iSCSI connection to the volume on the SAN and >>> a >>> >>>>>>>> >> > KVMPhysicalDisk is returned to be used in the attach logic >>> that >>> >>>>>>>> follows. >>> >>>>>>>> >> > >>> >>>>>>>> >> > When a volume is detached, getPhysicalDisk is invoked with >>> the >>> >>>>>>>> IQN of the >>> >>>>>>>> >> > volume if the storage pool in question is managed storage. >>> >>>>>>>> Otherwise, the >>> >>>>>>>> >> > normal vol.getPath() is used. >>> >>>>>>>> iScsiAdmStorageAdaptor.getPhysicalDisk just >>> >>>>>>>> >> > returns a new instance of KVMPhysicalDisk to be used in the >>> >>>>>>>> detach logic. >>> >>>>>>>> >> > >>> >>>>>>>> >> > Once the volume has been detached, >>> >>>>>>>> iScsiAdmStoragePool.deletePhysicalDisk >>> >>>>>>>> >> > is invoked if the storage pool is managed. >>> deletePhysicalDisk >>> >>>>>>>> removes the >>> >>>>>>>> >> > iSCSI connection to the volume using iscsiadm. >>> >>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>>> >> > On Sat, Sep 21, 2013 at 5:46 PM, Marcus Sorensen < >>> >>>>>>>> shadow...@gmail.com >>> >>>>>>>> >> >wrote: >>> >>>>>>>> >> > >>> >>>>>>>> >> >> Its the log4j properties file in /etc/cloudstack/agent >>> change >>> >>>>>>>> all INFO >>> >>>>>>>> >> to >>> >>>>>>>> >> >> DEBUG. I imagine the agent just isn't starting, you can >>> tail >>> >>>>>>>> the log >>> >>>>>>>> >> when >>> >>>>>>>> >> >> you try to start the service, or maybe it will spit >>> something >>> >>>>>>>> out into >>> >>>>>>>> >> one >>> >>>>>>>> >> >> of the other files in /var/log/cloudstack/agent >>> >>>>>>>> >> >> On Sep 21, 2013 5:19 PM, "Mike Tutkowski" < >>> >>>>>>>> mike.tutkow...@solidfire.com >>> >>>>>>>> >> > >>> >>>>>>>> >> >> wrote: >>> >>>>>>>> >> >> >>> >>>>>>>> >> >> > This is how I've been trying to query for the status of >>> the >>> >>>>>>>> service (I >>> >>>>>>>> >> >> > assume it could be started this way, as well, by changing >>> >>>>>>>> "status" to >>> >>>>>>>> >> >> > "start" or "restart"?): >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo >>> >>>>>>>> /usr/sbin/service >>> >>>>>>>> >> >> > cloudstack-agent status >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > I get this back: >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > Failed to execute: * could not access PID file for >>> >>>>>>>> cloudstack-agent >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > I've made a bunch of code changes recently, though, so I >>> >>>>>>>> think I'm >>> >>>>>>>> >> going >>> >>>>>>>> >> >> to >>> >>>>>>>> >> >> > rebuild and redeploy everything. >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > The debug info sounds helpful. Where can I set >>> enable.debug? >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > Thanks, Marcus! >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > On Sat, Sep 21, 2013 at 4:11 PM, Marcus Sorensen < >>> >>>>>>>> shadow...@gmail.com >>> >>>>>>>> >> >> > >wrote: >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > > OK, will check it out in the next few days. As >>> mentioned, >>> >>>>>>>> you can >>> >>>>>>>> >> set >>> >>>>>>>> >> >> up >>> >>>>>>>> >> >> > > your Ubuntu vm as the management server as well if all >>> >>>>>>>> else fails. >>> >>>>>>>> >> If >>> >>>>>>>> >> >> > you >>> >>>>>>>> >> >> > > can get to the mgmt server on 8250 from the KVM host, >>> then >>> >>>>>>>> you need >>> >>>>>>>> >> to >>> >>>>>>>> >> >> > > enable.debug on the agent. It won't run without >>> >>>>>>>> complaining loudly >>> >>>>>>>> >> if >>> >>>>>>>> >> >> it >>> >>>>>>>> >> >> > > can't get to the mgmt server, and I didn't see that in >>> >>>>>>>> your agent >>> >>>>>>>> >> log, >>> >>>>>>>> >> >> so >>> >>>>>>>> >> >> > > perhaps its not running. I assume you know how to >>> >>>>>>>> stop/start the >>> >>>>>>>> >> agent >>> >>>>>>>> >> >> on >>> >>>>>>>> >> >> > > KVM via 'service cloud stacks agent'. >>> >>>>>>>> >> >> > > On Sep 21, 2013 3:02 PM, "Mike Tutkowski" < >>> >>>>>>>> >> >> mike.tutkow...@solidfire.com> >>> >>>>>>>> >> >> > > wrote: >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > > > Hey Marcus, >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > I haven't yet been able to test my new code, but I >>> >>>>>>>> thought you >>> >>>>>>>> >> would >>> >>>>>>>> >> >> > be a >>> >>>>>>>> >> >> > > > good person to ask to review it: >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> >>> >>>>>>>> >> >>> >>>>>>>> >>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/ea74b312a8a36801994500407fd54f0cdda55e37 >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > All it is supposed to do is attach and detach a data >>> >>>>>>>> disk (that >>> >>>>>>>> >> has >>> >>>>>>>> >> >> > > > guaranteed IOPS) with KVM as the hypervisor. The data >>> >>>>>>>> disk >>> >>>>>>>> >> happens to >>> >>>>>>>> >> >> > be >>> >>>>>>>> >> >> > > > from SolidFire-backed storage - where we have a 1:1 >>> >>>>>>>> mapping >>> >>>>>>>> >> between a >>> >>>>>>>> >> >> > > > CloudStack volume and a data disk. >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > There is no support for hypervisor snapshots or stuff >>> >>>>>>>> like that >>> >>>>>>>> >> >> > (likely a >>> >>>>>>>> >> >> > > > future release)...just attaching and detaching a data >>> >>>>>>>> disk in 4.3. >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > Thanks! >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > On Fri, Sep 20, 2013 at 11:05 PM, Mike Tutkowski < >>> >>>>>>>> >> >> > > > mike.tutkow...@solidfire.com> wrote: >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > > When I re-deployed the DEBs, I didn't remove >>> >>>>>>>> cloudstack-agent >>> >>>>>>>> >> >> first. >>> >>>>>>>> >> >> > > > Would >>> >>>>>>>> >> >> > > > > that be a problem? I just did a sudo apt-get >>> install >>> >>>>>>>> >> >> > cloudstack-agent. >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > On Fri, Sep 20, 2013 at 11:03 PM, Mike Tutkowski < >>> >>>>>>>> >> >> > > > > mike.tutkow...@solidfire.com> wrote: >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > >> I get the same error running the command manually: >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >> mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo >>> >>>>>>>> >> /usr/sbin/service >>> >>>>>>>> >> >> > > > >> cloudstack-agent status >>> >>>>>>>> >> >> > > > >> * could not access PID file for cloudstack-agent >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >> On Fri, Sep 20, 2013 at 11:00 PM, Mike Tutkowski < >>> >>>>>>>> >> >> > > > >> mike.tutkow...@solidfire.com> wrote: >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >>> agent.log looks OK to me: >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,010 INFO >>> >>>>>>>> [cloud.agent.AgentShell] >>> >>>>>>>> >> >> (main:null) >>> >>>>>>>> >> >> > > > Agent >>> >>>>>>>> >> >> > > > >>> started >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,011 INFO >>> >>>>>>>> [cloud.agent.AgentShell] >>> >>>>>>>> >> >> (main:null) >>> >>>>>>>> >> >> > > > >>> Implementation Version is 4.3.0-SNAPSHOT >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,015 INFO >>> >>>>>>>> [cloud.agent.AgentShell] >>> >>>>>>>> >> >> (main:null) >>> >>>>>>>> >> >> > > > >>> agent.properties found at >>> >>>>>>>> >> /etc/cloudstack/agent/agent.properties >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,023 INFO >>> >>>>>>>> [cloud.agent.AgentShell] >>> >>>>>>>> >> >> (main:null) >>> >>>>>>>> >> >> > > > >>> Defaulting to using properties file for storage >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,024 INFO >>> >>>>>>>> [cloud.agent.AgentShell] >>> >>>>>>>> >> >> (main:null) >>> >>>>>>>> >> >> > > > >>> Defaulting to the constant time backoff algorithm >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,029 INFO >>> [cloud.utils.LogUtils] >>> >>>>>>>> >> (main:null) >>> >>>>>>>> >> >> > > log4j >>> >>>>>>>> >> >> > > > >>> configuration found at >>> >>>>>>>> /etc/cloudstack/agent/log4j-cloud.xml >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,163 INFO [cloud.agent.Agent] >>> >>>>>>>> (main:null) >>> >>>>>>>> >> id >>> >>>>>>>> >> >> > is 3 >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,197 INFO >>> >>>>>>>> >> >> > > > >>> [resource.virtualnetwork.VirtualRoutingResource] >>> >>>>>>>> (main:null) >>> >>>>>>>> >> >> > > > >>> VirtualRoutingResource _scriptDir to use: >>> >>>>>>>> >> >> scripts/network/domr/kvm >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> However, I wasn't aware that setup.log was >>> >>>>>>>> important. This >>> >>>>>>>> >> seems >>> >>>>>>>> >> >> to >>> >>>>>>>> >> >> > > be >>> >>>>>>>> >> >> > > > a >>> >>>>>>>> >> >> > > > >>> problem, but I'm not sure what it might indicate: >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service >>> >>>>>>>> cloudstack-agent >>> >>>>>>>> >> status >>> >>>>>>>> >> >> > > > >>> DEBUG:root:Failed to execute: * could not access >>> PID >>> >>>>>>>> file for >>> >>>>>>>> >> >> > > > >>> cloudstack-agent >>> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service >>> >>>>>>>> cloudstack-agent >>> >>>>>>>> >> start >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> On Fri, Sep 20, 2013 at 10:53 PM, Marcus >>> Sorensen < >>> >>>>>>>> >> >> > > shadow...@gmail.com >>> >>>>>>>> >> >> > > > >wrote: >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>>> Sorry, I saw that in the log, I thought it was >>> the >>> >>>>>>>> agent log >>> >>>>>>>> >> for >>> >>>>>>>> >> >> > > some >>> >>>>>>>> >> >> > > > >>>> reason. Is the agent started? That might be the >>> >>>>>>>> place to >>> >>>>>>>> >> look. >>> >>>>>>>> >> >> > There >>> >>>>>>>> >> >> > > > is >>> >>>>>>>> >> >> > > > >>>> an >>> >>>>>>>> >> >> > > > >>>> agent log for the agent and one for the setup >>> when >>> >>>>>>>> it adds >>> >>>>>>>> >> the >>> >>>>>>>> >> >> > host, >>> >>>>>>>> >> >> > > > >>>> both >>> >>>>>>>> >> >> > > > >>>> in /var/log >>> >>>>>>>> >> >> > > > >>>> On Sep 20, 2013 10:42 PM, "Mike Tutkowski" < >>> >>>>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com> >>> >>>>>>>> >> >> > > > >>>> wrote: >>> >>>>>>>> >> >> > > > >>>> >>> >>>>>>>> >> >> > > > >>>> > Is it saying that the MS is at the IP address >>> or >>> >>>>>>>> the KVM >>> >>>>>>>> >> host? >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > The KVM host is at 192.168.233.10. >>> >>>>>>>> >> >> > > > >>>> > The MS host is at 192.168.233.1. >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > I see this for my host Global Settings >>> parameter: >>> >>>>>>>> >> >> > > > >>>> > hostThe ip address of management >>> >>>>>>>> server192.168.233.1 >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > /etc/cloudstack/agent/agent.properties has a >>> >>>>>>>> >> >> host=192.168.233.1 >>> >>>>>>>> >> >> > > > value. >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > On Fri, Sep 20, 2013 at 10:21 PM, Marcus >>> Sorensen >>> >>>>>>>> < >>> >>>>>>>> >> >> > > > >>>> shadow...@gmail.com >>> >>>>>>>> >> >> > > > >>>> > >wrote: >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > > The log says your mgmt server is >>> >>>>>>>> 192.168.233.10? But you >>> >>>>>>>> >> >> tried >>> >>>>>>>> >> >> > > to >>> >>>>>>>> >> >> > > > >>>> telnet >>> >>>>>>>> >> >> > > > >>>> > to >>> >>>>>>>> >> >> > > > >>>> > > 192.168.233.1? It might be enough to change >>> >>>>>>>> that in >>> >>>>>>>> >> >> > > > >>>> > > /etc/cloudstack/agent/agent.properties, but >>> you >>> >>>>>>>> may want >>> >>>>>>>> >> to >>> >>>>>>>> >> >> > edit >>> >>>>>>>> >> >> > > > the >>> >>>>>>>> >> >> > > > >>>> > config >>> >>>>>>>> >> >> > > > >>>> > > as well to tell it the real ms IP. >>> >>>>>>>> >> >> > > > >>>> > > On Sep 20, 2013 10:12 PM, "Mike Tutkowski" < >>> >>>>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > wrote: >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > > Here's what my /etc/network/interfaces >>> file >>> >>>>>>>> looks >>> >>>>>>>> >> like, if >>> >>>>>>>> >> >> > > that >>> >>>>>>>> >> >> > > > >>>> is of >>> >>>>>>>> >> >> > > > >>>> > > > interest (the 192.168.233.0 network is the >>> >>>>>>>> NAT network >>> >>>>>>>> >> >> > VMware >>> >>>>>>>> >> >> > > > >>>> Fusion >>> >>>>>>>> >> >> > > > >>>> > set >>> >>>>>>>> >> >> > > > >>>> > > > up): >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > auto lo >>> >>>>>>>> >> >> > > > >>>> > > > iface lo inet loopback >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > auto eth0 >>> >>>>>>>> >> >> > > > >>>> > > > iface eth0 inet manual >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > auto cloudbr0 >>> >>>>>>>> >> >> > > > >>>> > > > iface cloudbr0 inet static >>> >>>>>>>> >> >> > > > >>>> > > > address 192.168.233.10 >>> >>>>>>>> >> >> > > > >>>> > > > netmask 255.255.255.0 >>> >>>>>>>> >> >> > > > >>>> > > > network 192.168.233.0 >>> >>>>>>>> >> >> > > > >>>> > > > broadcast 192.168.233.255 >>> >>>>>>>> >> >> > > > >>>> > > > dns-nameservers 8.8.8.8 >>> >>>>>>>> >> >> > > > >>>> > > > bridge_ports eth0 >>> >>>>>>>> >> >> > > > >>>> > > > bridge_fd 5 >>> >>>>>>>> >> >> > > > >>>> > > > bridge_stp off >>> >>>>>>>> >> >> > > > >>>> > > > bridge_maxwait 1 >>> >>>>>>>> >> >> > > > >>>> > > > post-up route add default gw >>> >>>>>>>> 192.168.233.2 metric 1 >>> >>>>>>>> >> >> > > > >>>> > > > pre-down route del default gw >>> >>>>>>>> 192.168.233.2 >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > On Fri, Sep 20, 2013 at 10:08 PM, Mike >>> >>>>>>>> Tutkowski < >>> >>>>>>>> >> >> > > > >>>> > > > mike.tutkow...@solidfire.com> wrote: >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > > You appear to be correct. This is from >>> the >>> >>>>>>>> MS log >>> >>>>>>>> >> >> (below). >>> >>>>>>>> >> >> > > > >>>> Discovery >>> >>>>>>>> >> >> > > > >>>> > > > timed >>> >>>>>>>> >> >> > > > >>>> > > > > out. >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I'm not sure why this would be. My >>> network >>> >>>>>>>> settings >>> >>>>>>>> >> >> > > shouldn't >>> >>>>>>>> >> >> > > > >>>> have >>> >>>>>>>> >> >> > > > >>>> > > > changed >>> >>>>>>>> >> >> > > > >>>> > > > > since the last time I tried this. >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I am able to ping the KVM host from the >>> MS >>> >>>>>>>> host and >>> >>>>>>>> >> vice >>> >>>>>>>> >> >> > > > versa. >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I'm even able to manually kick off a VM >>> on >>> >>>>>>>> the KVM >>> >>>>>>>> >> host >>> >>>>>>>> >> >> > and >>> >>>>>>>> >> >> > > > >>>> ping from >>> >>>>>>>> >> >> > > > >>>> > > it >>> >>>>>>>> >> >> > > > >>>> > > > > to the MS host. >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I am using NAT from my Mac OS X host >>> (also >>> >>>>>>>> running >>> >>>>>>>> >> the >>> >>>>>>>> >> >> CS >>> >>>>>>>> >> >> > > MS) >>> >>>>>>>> >> >> > > > >>>> to the >>> >>>>>>>> >> >> > > > >>>> > VM >>> >>>>>>>> >> >> > > > >>>> > > > > running KVM (VMware Fusion). >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,141 DEBUG >>> >>>>>>>> >> >> > > > >>>> [c.c.h.k.d.LibvirtServerDiscoverer] >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >>> :ctx-6b28dc48) >>> >>>>>>>> Timeout, >>> >>>>>>>> >> to >>> >>>>>>>> >> >> > wait >>> >>>>>>>> >> >> > > > for >>> >>>>>>>> >> >> > > > >>>> the >>> >>>>>>>> >> >> > > > >>>> > > host >>> >>>>>>>> >> >> > > > >>>> > > > > connecting to mgt svr, assuming it is >>> failed >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,144 WARN >>> >>>>>>>> >> >> [c.c.r.ResourceManagerImpl] >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >>> :ctx-6b28dc48) >>> >>>>>>>> Unable to >>> >>>>>>>> >> >> find >>> >>>>>>>> >> >> > > the >>> >>>>>>>> >> >> > > > >>>> server >>> >>>>>>>> >> >> > > > >>>> > > > > resources at http://192.168.233.10 >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,145 INFO >>> >>>>>>>> >> >> > [c.c.u.e.CSExceptionErrorCode] >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >>> :ctx-6b28dc48) >>> >>>>>>>> Could not >>> >>>>>>>> >> >> find >>> >>>>>>>> >> >> > > > >>>> exception: >>> >>>>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException >>> in >>> >>>>>>>> error code >>> >>>>>>>> >> >> list >>> >>>>>>>> >> >> > > for >>> >>>>>>>> >> >> > > > >>>> > > exceptions >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,147 WARN >>> >>>>>>>> >> [o.a.c.a.c.a.h.AddHostCmd] >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 >>> :ctx-6b28dc48) >>> >>>>>>>> Exception: >>> >>>>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException: >>> >>>>>>>> Unable to add >>> >>>>>>>> >> >> the >>> >>>>>>>> >> >> > > host >>> >>>>>>>> >> >> > > > >>>> > > > > at >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> >>> >>>>>>>> >> >>> >>>>>>>> >>> com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:778) >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I do seem to be able to telnet in from >>> my >>> >>>>>>>> KVM host to >>> >>>>>>>> >> >> the >>> >>>>>>>> >> >> > MS >>> >>>>>>>> >> >> > > > >>>> host's >>> >>>>>>>> >> >> > > > >>>> > > 8250 >>> >>>>>>>> >> >> > > > >>>> > > > > port: >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > mtutkowski@ubuntu:~$ telnet >>> 192.168.233.1 >>> >>>>>>>> 8250 >>> >>>>>>>> >> >> > > > >>>> > > > > Trying 192.168.233.1... >>> >>>>>>>> >> >> > > > >>>> > > > > Connected to 192.168.233.1. >>> >>>>>>>> >> >> > > > >>>> > > > > Escape character is '^]'. >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > -- >>> >>>>>>>> >> >> > > > >>>> > > > *Mike Tutkowski* >>> >>>>>>>> >> >> > > > >>>> > > > *Senior CloudStack Developer, SolidFire >>> Inc.* >>> >>>>>>>> >> >> > > > >>>> > > > e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > >>>> > > > o: 303.746.7302 >>> >>>>>>>> >> >> > > > >>>> > > > Advancing the way the world uses the >>> >>>>>>>> >> >> > > > >>>> > > > cloud< >>> >>>>>>>> >> http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> >> >> > > > >>>> > > > *™* >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > -- >>> >>>>>>>> >> >> > > > >>>> > *Mike Tutkowski* >>> >>>>>>>> >> >> > > > >>>> > *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> >> > > > >>>> > e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > >>>> > o: 303.746.7302 >>> >>>>>>>> >> >> > > > >>>> > Advancing the way the world uses the >>> >>>>>>>> >> >> > > > >>>> > cloud< >>> >>>>>>>> http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> >> >> > > > >>>> > *™* >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >>> -- >>> >>>>>>>> >> >> > > > >>> *Mike Tutkowski* >>> >>>>>>>> >> >> > > > >>> *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> >> > > > >>> e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > >>> o: 303.746.7302 >>> >>>>>>>> >> >> > > > >>> Advancing the way the world uses the cloud< >>> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> >> >> > > > >>> *™* >>> >>>>>>>> >> >> > > > >>> >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > >> -- >>> >>>>>>>> >> >> > > > >> *Mike Tutkowski* >>> >>>>>>>> >> >> > > > >> *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> >> > > > >> e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > >> o: 303.746.7302 >>> >>>>>>>> >> >> > > > >> Advancing the way the world uses the cloud< >>> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> >> >> > > > >> *™* >>> >>>>>>>> >> >> > > > >> >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > -- >>> >>>>>>>> >> >> > > > > *Mike Tutkowski* >>> >>>>>>>> >> >> > > > > *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> >> > > > > e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > > o: 303.746.7302 >>> >>>>>>>> >> >> > > > > Advancing the way the world uses the cloud< >>> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> >> >> > > > > *™* >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > -- >>> >>>>>>>> >> >> > > > *Mike Tutkowski* >>> >>>>>>>> >> >> > > > *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> >> > > > e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > > > o: 303.746.7302 >>> >>>>>>>> >> >> > > > Advancing the way the world uses the >>> >>>>>>>> >> >> > > > cloud< >>> http://solidfire.com/solution/overview/?video=play >>> >>>>>>>> > >>> >>>>>>>> >> >> > > > *™* >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > -- >>> >>>>>>>> >> >> > *Mike Tutkowski* >>> >>>>>>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> >> > e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> >> > o: 303.746.7302 >>> >>>>>>>> >> >> > Advancing the way the world uses the >>> >>>>>>>> >> >> > cloud<http://solidfire.com/solution/overview/?video=play >>> > >>> >>>>>>>> >> >> > *™* >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> >>> >>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>>> >> > -- >>> >>>>>>>> >> > *Mike Tutkowski* >>> >>>>>>>> >> > *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> >> > e: mike.tutkow...@solidfire.com >>> >>>>>>>> >> > o: 303.746.7302 >>> >>>>>>>> >> > Advancing the way the world uses the >>> >>>>>>>> >> > cloud<http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> >> > *™* >>> >>>>>>>> >> >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > -- >>> >>>>>>>> > *Mike Tutkowski* >>> >>>>>>>> > *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>>> > e: mike.tutkow...@solidfire.com >>> >>>>>>>> > o: 303.746.7302 >>> >>>>>>>> > Advancing the way the world uses the >>> >>>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play> >>> >>>>>>>> > *™* >>> >>>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> -- >>> >>>>>>> *Mike Tutkowski* >>> >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>>> e: mike.tutkow...@solidfire.com >>> >>>>>>> o: 303.746.7302 >>> >>>>>>> Advancing the way the world uses the cloud< >>> http://solidfire.com/solution/overview/?video=play> >>> >>>>>>> *™* >>> >>>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> -- >>> >>>>>> *Mike Tutkowski* >>> >>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>>> e: mike.tutkow...@solidfire.com >>> >>>>>> o: 303.746.7302 >>> >>>>>> Advancing the way the world uses the cloud< >>> http://solidfire.com/solution/overview/?video=play> >>> >>>>>> *™* >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> *Mike Tutkowski* >>> >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>> >>>>> e: mike.tutkow...@solidfire.com >>> >>>>> o: 303.746.7302 >>> >>>>> Advancing the way the world uses the cloud< >>> http://solidfire.com/solution/overview/?video=play> >>> >>>>> *™* >>> >>>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> *Mike Tutkowski* >>> >>>> *Senior CloudStack Developer, SolidFire Inc.* >>> >>>> e: mike.tutkow...@solidfire.com >>> >>>> o: 303.746.7302 >>> >>>> Advancing the way the world uses the cloud< >>> http://solidfire.com/solution/overview/?video=play> >>> >>>> *™* >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> *Mike Tutkowski* >>> >>> *Senior CloudStack Developer, SolidFire Inc.* >>> >>> e: mike.tutkow...@solidfire.com >>> >>> o: 303.746.7302 >>> >>> Advancing the way the world uses the cloud< >>> http://solidfire.com/solution/overview/?video=play> >>> >>> *™* >>> >>> >>> >> >>> >> >>> >> >>> >> -- >>> >> *Mike Tutkowski* >>> >> *Senior CloudStack Developer, SolidFire Inc.* >>> >> e: mike.tutkow...@solidfire.com >>> >> o: 303.746.7302 >>> >> Advancing the way the world uses the cloud< >>> http://solidfire.com/solution/overview/?video=play> >>> >> *™* >>> >> >>> > >>> > >>> > >>> > -- >>> > *Mike Tutkowski* >>> > *Senior CloudStack Developer, SolidFire Inc.* >>> > e: mike.tutkow...@solidfire.com >>> > o: 303.746.7302 >>> > Advancing the way the world uses the >>> > cloud<http://solidfire.com/solution/overview/?video=play> >>> > *™* >>> >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the >> cloud<http://solidfire.com/solution/overview/?video=play> >> *™* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™*
