Hi Prasanna and all, I managed to work out on how to assign public IP to the VM instance using static NAT.
However, I am still stuck with the console proxy time-out issue. It seems that my problem is exactly the one mentioned on this bug reported to JIRA: https://issues.apache.org/jira/browse/CLOUDSTACK-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel May I know whether there's any resolution to the bug above? The issue seems to be identical to the one I am facing right now. Any other way on how to console in to the newly created VM other than using the console proxy method? Unlike system VM, the instance VM doesn't give any link local address which I can SSH directly from the KVM host. Am I able to using virsh command to log in to the instance? Looking forward to your reply, thank you. Cheers. On Mon, Jul 15, 2013 at 3:19 PM, Indra Pramana <in...@sg.or.id> wrote: > Hi Prasanna and all, > > I managed to create a VM instance using the NFS primary storage (not using > the RBD primary storage due to the bug we discussed on another thread). > > However, there is no public IP being assigned to the VM instance (unlike > the system VMs which are getting public IPs). How do I assign public IP to > the instance? > > === > NIC 1 (Default) > Network Name Indra > IP Address 10.19.8.186 > Type Isolated > Gateway 10.19.0.1 > Netmask 255.255.240.0 > Is Default Yes > === > > The network was created under the > "DefaultIsolatedNetworkOfferingWithSourceNatService" network offering. I am > not able to create the network using different network offerings, despite > the other offerings are available under Service Offerings > Network > Offerings and the state of those network offerings are enabled. Is it > related to the way how I configured the network bridges and configuration > earlier during zone creation, anything that I might have missed out? > > I also tried to "view console" to the VM (via the web GUI) and not able to > do so. Earlier I got the "Unable to start console session as connection is > refused by the machine you are accessing" error message, which I have fixed > by changing the /etc/libvirt/qemu.conf to enable this line: > > vnc_listen = "0.0.0.0" > > because initially the VNC only listens on localhost (127.0.0.1) by > default. Now I am no longer getting the error message, but the console > window still failed to load after some time, with Firefox browser error > message: > > === > The connection was reset > > The connection to the server was reset while the page was loading. > === > > How can I further troubleshoot on this? I tried to SSH into the console > proxy VM and can't find anything useful on the logs there. > > > Looking forward to your reply, thank you. > > Cheers. > > > > On Mon, Jul 15, 2013 at 12:38 PM, Indra Pramana <in...@sg.or.id> wrote: > >> Hi Prasanna, >> >> On Mon, Jul 15, 2013 at 12:20 PM, Prasanna Santhanam <t...@apache.org>wrote: >> >>> On Sat, Jul 13, 2013 at 02:25:35PM +0800, Indra Pramana wrote: >>> > Hi Prasanna, >>> > >>> > Good day to you, and thank you for your e-mail. >>> > >>> Were you able to get beyond this error? >>> >> >> Yes. I managed to get around the network bridges issue by setting up this >> on my KVM host: >> >> === >> auto eth0.5 >> iface eth0.5 inet static >> address X.X.X.22 >> netmask 255.255.255.240 >> network X.X.X.16 >> broadcast X.X.X.31 >> gateway X.X.X.17 >> # dns-* options are implemented by the resolvconf package, if >> installed >> dns-nameservers 8.8.8.8 8.8.4.4 >> dns-search xxx.com >> >> # Public network >> auto cloudbr0 >> iface cloudbr0 inet manual >> bridge_ports eth1.201 >> bridge_fd 5 >> bridge_stp off >> bridge_maxwait 1 >> >> # Guest network >> auto cloudbr1 >> iface cloudbr1 inet manual >> bridge_ports eth1.41 >> bridge_fd 5 >> bridge_stp off >> bridge_maxwait 1 >> >> # Management network >> auto cloudbr2 >> iface cloudbr2 inet manual >> bridge_ports eth0.6 >> bridge_fd 5 >> bridge_stp off >> bridge_maxwait 1 >> >> # The 10G network interface >> auto eth2 >> iface eth2 inet static >> address 10.237.11.22 >> netmask 255.255.255.0 >> network 10.237.11.0 >> broadcast 10.237.11.255 >> === >> >> On the Cloudstack GUI, I created 3 physical networks: >> >> cloudbr0 - public network >> cloudbr1 - guest network >> cloudbr2 - management (and storage) network >> >> However, when I tried to setup the public network using VLAN 201, it >> failed because CloudStack tried to create a new bridge (i.e. breth1-201) >> instead of using the existing bridge I have set up for public network >> (cloudbr mapped to eth1.201). So I set up the public network using a >> different VLAN (VLAN 21) and change the settings on my router's gateway to >> use VLAN 21 instead of VLAN 201. >> >> === >> root@hv-kvm-02:~# brctl show >> bridge name bridge id STP enabled interfaces >> breth1-21 8000.002590c29d7d no eth1.21 >> vnet2 >> vnet5 >> cloud0 8000.fe00a9fe0165 no vnet0 >> vnet3 >> cloudbr0 8000.002590c29d7d no eth1.201 >> cloudbr1 8000.002590c29d7d no eth1.41 >> cloudbr2 8000.002590c29d7c no eth0.6 >> vnet1 >> vnet4 >> vnet6 >> virbr0 8000.000000000000 yes >> === >> >> The above setup works, but I am not too sure whether this is the correct >> way on how to do it. Can you advise? >> >> Looking forward to your reply, thank you. >> >> Cheers. >> >> >> >> >> >>> > Yesterday I troubleshooted further and found out that the mapping of >>> the >>> > bridge is incorrect. For example, eth1 of the SSVM, which is supposed >>> to be >>> > private network interface for communication with management server, is >>> > being mapped to cloudbr0 bridge which is used for public. While eth2 >>> of the >>> > SSVM, which is supposed to be public, is being mapped to cloudbr1 >>> bridge >>> > which is supposed for guest traffic. I checked with brctl show on the >>> KVM >>> > host and traced the MAC address of the SSVM's eth interfaces. >>> > >>> > I have tried different type of configuration on the KVM hosts' >>> > /etc/network/interfaces (I am using Ubuntu), different ways on how to >>> > configure the advanced network zone (physical network interface and KVM >>> > label for each traffic), but I am still not able to get it working. >>> > >>> > Can please advise what is the best way to set-up the network bridges? >>> >>> Use brctl to add bridges. That's how I've done it. >>> >>> > >>> > My /etc/network/interfaces on KVM host: >>> > >>> > http://pastebin.com/nx8xJ1L2 >>> > >>> > I used 3 physical NICs: >>> > >>> > eth0 --> for management and secondary storage traffic >>> > eth1 --> for public and guest traffic >>> > eth2 --> for primary storage traffic to the Ceph RBD (separate NIC not >>> > configured by CloudStack) >>> > >>> > On the Cloudstack GUI, while creating the zone, I used 3 physical >>> networks: >>> > >>> > eth0.6 --> Management traffic (KVM label: eth0.6) >>> > cloudbr0 --> Public traffic (KVM label: cloudbr0) >>> > cloudbr1 --> Guest traffic (KVM label: cloudbr1) >>> > >>> Is eth0.6 on a tagged VLAN? >>> >>> > I didn't specify storage traffic since I presume it will use the >>> management >>> > VLAN. >>> >>> That's correct. >>> >>> > >>> > However, I always failed to add the KVM host using the above >>> configuration. >>> > I tried to run cloudstack-setup-agent manually on the KVM host and >>> getting >>> > this error message: >>> > >>> > root@hv-kvm-02:/etc/cloudstack/agent# cloudstack-setup-agent -m >>> x.x.x.18 -z >>> > 3 -p 3 -c 3 -g 00dd5dba-7419-3689-bc51-1671035c0d8f -a >>> --pubNic=cloudbr0 >>> > --prvNic=eth0.6 --guestNic=cloudbr1 >>> > Starting to configure your system: >>> > Configure Apparmor ... [OK] >>> > Configure Network ... [Failed] >>> > eth0.6 is not a bridge >>> > Try to restore your system: >>> > Restore Apparmor ... [OK] >>> > Restore Network ... [OK] >>> > >>> > When I tried to create another bridge called cloudbr2 with >>> bridge_ports to >>> > eth0.6, the KVM host's network didn't work. >>> > >>> > Can advise what is the best practice on configuring the network >>> bridges on >>> > the advanced network zone setup? >>> > >>> > Looking forward to your reply, thank you. >>> > >>> > Cheers. >>> > >>> > >>> > >>> > On Sat, Jul 13, 2013 at 12:28 PM, Prasanna Santhanam <t...@apache.org> >>> wrote: >>> > >>> > > See Inline, >>> > > >>> > > On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote: >>> > > > Hi Wido, >>> > > > >>> > > > Noted, can't wait for 4.2 to be released. :) >>> > > > >>> > > > Dear Prasanna, Wido and all, >>> > > > >>> > > > I just realised that while the system VMs are running, they are >>> still not >>> > > > accessible through the public IPs assigned to them. I have been >>> waiting >>> > > for >>> > > > the SSVM to download the default CentOS template and it doesn't >>> appear on >>> > > > the template list. >>> > > > >>> > > > I tried to SSH into the SSVM via the local link address from the >>> KVM >>> > > host, >>> > > > and running the health check >>> /usr/local/cloud/systemvm/ssvm-check.sh >>> > > shows >>> > > > that the VM cannot reach anywhere. It cannot reach the public DNS >>> server >>> > > (I >>> > > > used Google's 8.8.8.8), cannot reach the management server, and >>> cannot >>> > > even >>> > > > reach the public IP gateway. >>> > > >>> > > Ok - we're not there yet. But there's a few troubleshooting steps you >>> > > can try: >>> > > >>> > > 1. Which of the checks fail apart from the DNS ping? >>> > > 2. Are you able to have those tests pass from the KVM host itself? >>> > > 3. Can you paste the output of $route -n on your SSVM and on your >>> > > host? >>> > > >>> > > > >>> > > > Is it due to misconfiguration of the KVM network bridges? How can >>> I see >>> > > the >>> > > > mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 >>> and >>> > > eth3) >>> > > > and the actual physical NIC interfaces on the KVM hosts (eth0) and >>> the >>> > > > network bridges (cloudbr0, cloudbr1)? Any logs I can verify to >>> ensure >>> > > that >>> > > > the VLAN and network bridging is working? >>> > > > >>> > > >>> > > You can see this from the dumpxml. >>> > > >>> > > $virsh list >>> > > (gives you all the domains. find the ssvm for example) >>> > > >>> > > $virsh dumpxml my-domain > mydomain.xml >>> > > >>> > > > Appreciate any advice. >>> > > > >>> > > > Thank you. >>> > > > >>> > > > >>> > > > >>> > > > On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander < >>> w...@widodh.nl> >>> > > wrote: >>> > > > >>> > > > > On 07/12/2013 10:14 AM, Indra Pramana wrote: >>> > > > > >>> > > > >> Hi Prasanna, >>> > > > >> >>> > > > >> I managed to fix the problem, thanks for your advice to turn >>> the agent >>> > > > >> log level to debug: >>> > > > >> >>> > > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/** >>> > > > >> KVM+agent+debug< >>> > > >>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug> >>> > > > >> >>> > > > >> From the log, I found out that the agent on the KVM host tried >>> to NFS >>> > > > >> mount directly to 103.25.200.19:/mnt/vol1/sec-** >>> > > > >> storage/template/tmpl/1/3, >>> > > > >> which was not allowed by the NFS server due to its default >>> > > configuration >>> > > > >> to only allow to mount to /mnt/vol1/sec-storage (the root of >>> the NFS >>> > > > >> share). >>> > > > >> >>> > > > >> >>> > > > > Ah, that's odd! >>> > > > > >>> > > > > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so >>> that >>> > > > > limitation will be gone. >>> > > > > >>> > > > > Wido >>> > > > > >>> > > > > After I changed the NFS server configuration to allow mount to >>> all >>> > > > >> sub-directories, re-export the NFS and voila, the system was >>> able to >>> > > > >> download the template and now both the system VMs (CPVM and >>> SSVM) are >>> > > > >> running! >>> > > > >> >>> > > > >> Many thanks for your help! :) >>> > > > >> >>> > > > >> Cheers. >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <in...@sg.or.id >>> > > > >> <mailto:in...@sg.or.id>> wrote: >>> > > > >> >>> > > > >> Hi Prasanna, >>> > > > >> >>> > > > >> Good day to you, and thank you for your e-mail. >>> > > > >> >>> > > > >> Yes, the cloudstack-agent service is running on both the KVM >>> > > hosts. >>> > > > >> There is no "cloud" user being created though, when I >>> installed >>> > > the >>> > > > >> agent. I installed the agent as root. >>> > > > >> >>> > > > >> root@hv-kvm-01:/home/indra# service cloudstack-agent status >>> > > > >> * cloud-agent is running >>> > > > >> >>> > > > >> root@hv-kvm-01:/home/indra# su - cloud >>> > > > >> Unknown id: cloud >>> > > > >> >>> > > > >> Please advise how can I resolve this problem, shall I >>> create the >>> > > > >> Unix "cloud" user manually? Basically I follow this >>> instruction to >>> > > > >> prepare the KVM host and install the CloudStack agent: >>> > > > >> >>> > > > >> >>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/** >>> > > > >> >>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html< >>> > > >>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html >>> > > > >>> > > > >> >>> > > > >> with this instruction from Wido on how to prepare libvirt >>> with >>> > > Ceph >>> > > > >> RBD storage pool support: >>> > > > >> >>> > > > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-** >>> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/< >>> > > >>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/ >>> > > > >>> > > > >> >>> > > > >> I also have checked /var/log/cloud/agent/agent.log and I >>> don't see >>> > > > >> any error messages, except this error message which will >>> show up >>> > > > >> every time I restart the agent: >>> > > > >> >>> > > > >> 2013-07-12 15:22:47,454 ERROR >>> > > [cloud.resource.**ServerResourceBase] >>> > > > >> (main:null) Nics are not configured! >>> > > > >> 2013-07-12 15:22:47,459 INFO >>> > > [cloud.resource.**ServerResourceBase] >>> > > > >> (main:null) Designating private to be nic eth0.5 >>> > > > >> >>> > > > >> More logs can be found here: http://pastebin.com/yeNmCt7S >>> > > > >> >>> > > > >> I have configured the network bridges on the NIC interface >>> as per >>> > > > >> this instruction: >>> > > > >> >>> > > > >> >>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/** >>> > > > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.** >>> > > > >> html#hypervisor-host-install-**network< >>> > > >>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network >>> > > > >>> > > > >> >>> > > > >> On the zone, I used advanced network configuration with >>> just one >>> > > > >> physical network for management, public and guest/private. I >>> > > didn't >>> > > > >> include storage, which by default the traffic will use the >>> > > > >> management VLAN network. >>> > > > >> >>> > > > >> Please advise if there's anything else I might have been >>> missing. >>> > > > >> >>> > > > >> Looking forward to your reply, thank you. >>> > > > >> >>> > > > >> Cheers. >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam < >>> > > t...@apache.org >>> > > > >> <mailto:t...@apache.org>> wrote: >>> > > > >> >>> > > > >> Indeed, cloudstack will go through the allocation to >>> startup >>> > > the >>> > > > >> system VMs too. So that process is failing to recognize >>> the >>> > > volume >>> > > > >> (.qcow2) present on your NFS storage. >>> > > > >> >>> > > > >> Can you check if your cloudstack agent service is >>> running with >>> > > > >> the KVM >>> > > > >> host? And it should've created the user cloud. $id >>> cloud to >>> > > check >>> > > > >> if >>> > > > >> there is a user. >>> > > > >> >>> > > > >> Did you see what's happening in the agent logs? These >>> are >>> > > under >>> > > > >> /var/log/cloud/ on your host when the systemVMs are >>> coming up. >>> > > > >> If the >>> > > > >> logs are not showing any useful information you can >>> turn on >>> > > debug >>> > > > >> level for more verbosity. >>> > > > >> >>> > > > >> See here: >>> https://cwiki.apache.org/**confluence/x/FgPMAQ< >>> > > https://cwiki.apache.org/confluence/x/FgPMAQ> >>> > > > >> >>> > > > >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana >>> wrote: >>> > > > >> > Hi Prasanna, >>> > > > >> > >>> > > > >> > Good day to you, and thank you for your e-mail. >>> > > > >> > >>> > > > >> > Yes, when I export the NFS, I set the permission so >>> that >>> > > > >> normal user will >>> > > > >> > be able to have read/write access to the files >>> > > > >> (no_root_squash). >>> > > > >> > >>> > > > >> > I have tested and I can have read/write access from >>> my KVM >>> > > > >> hosts using >>> > > > >> > normal user. BTW, there's no "cloud" user on the >>> hosts, I >>> > > > >> believe it's not >>> > > > >> > created during cloudstack-agent installation? >>> > > > >> > >>> > > > >> > In any case, do you think the template issue and the >>> > > storage >>> > > > >> pool >>> > > > >> > allocation issue might be related, or are they two >>> > > different >>> > > > >> problems >>> > > > >> > altogether? >>> > > > >> > >>> > > > >> > Looking forward to your reply, thank you. >>> > > > >> > >>> > > > >> > Cheers. >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam >>> > > > >> <t...@apache.org <mailto:t...@apache.org>> wrote: >>> > > > >> > >>> > > > >> > > Can you access the file as user root? Or user >>> cloud? The >>> > > > >> cloudstack >>> > > > >> > > agent on your KVM host runs as user cloud and the >>> NFS >>> > > > >> permissions >>> > > > >> > > might be disallowing the volume (.qcow2) from being >>> > > accessed. >>> > > > >> > > >>> > > > >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra >>> Pramana >>> > > > >> wrote: >>> > > > >> > > > Hi Prasanna, >>> > > > >> > > > >>> > > > >> > > > Good day to you, and thank you for your e-mail. >>> > > > >> > > > >>> > > > >> > > > Yes, the file exists. I can access the file from >>> the >>> > > > >> management server >>> > > > >> > > and >>> > > > >> > > > the two hypervisors hosts if I mount manually. >>> > > > >> > > > >>> > > > >> > > > [root@cs-nas-01 >>> /mnt/vol1/sec-storage/**template/tmpl/1/3]# >>> > > > >> ls >>> > > > >> > > > -la >>> > > > >> > > > total >>> > > > >> > > > 1418787 >>> > > > >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21 >>> > > > >> > > > . >>> > > > >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17 >>> > > > >> > > > .. >>> > > > >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21 >>> > > > >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2 >>> > > > >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21 >>> > > > >> > > > template.properties >>> > > > >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/** >>> > > > >> template/tmpl/1/3]# >>> > > > >> > > > pwd >>> > > > >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3 >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > Any advise? >>> > > > >> > > > >>> > > > >> > > > Looking forward to your reply, thank you. >>> > > > >> > > > >>> > > > >> > > > Cheers. >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna >>> Santhanam >>> > > > >> <t...@apache.org <mailto:t...@apache.org>> >>> > > > >> >>> > > > >> > > wrote: >>> > > > >> > > > >>> > > > >> > > > > Can you check whether there is a file at: >>> > > > >> > > > > >>> > > > >> nfs:// >>> 103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/ >>> > > <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/> >>> > > > >> < >>> > > http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/< >>> > > http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/> >>> > > > >> > >>> > > > >> >>> > > > >> > > > > >>> > > > >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra >>> > > Pramana >>> > > > >> wrote: >>> > > > >> > > > > > Hi Prasanna, >>> > > > >> > > > > > >>> > > > >> > > > > > Thanks for your e-mail. >>> > > > >> > > > > > >>> > > > >> > > > > > I have tried restarting the management >>> server, and >>> > > > >> the problem still >>> > > > >> > > > > > persists. I even tried to re-do the >>> installation >>> > > and >>> > > > >> configuration >>> > > > >> > > again >>> > > > >> > > > > > from scratch last night, but the problem >>> still >>> > > there. >>> > > > >> > > > > > >>> > > > >> > > > > > I also noted that on the beginning of the >>> logs, I >>> > > > >> found some error >>> > > > >> > > > > messages >>> > > > >> > > > > > saying that the template cannot be >>> downloaded to >>> > > the >>> > > > >> pool. See this >>> > > > >> > > logs: >>> > > > >> > > > > > >>> > > > >> > > > > > http://pastebin.com/BY1AVJ08 >>> > > > >> > > > > > >>> > > > >> > > > > > It says it failed because cannot get volume >>> from >>> > > the >>> > > > >> pool. Could it >>> > > > >> > > be >>> > > > >> > > > > > related, i.e. the absence of the template >>> caused >>> > > the >>> > > > >> system VMs >>> > > > >> > > cannot be >>> > > > >> > > > > > created and started? >>> > > > >> > > > > > >>> > > > >> > > > > > I have ensured that I downloaded the system >>> VM >>> > > > >> template using >>> > > > >> > > > > > cloud-install-sys-tmplt and verified that the >>> > > > >> template is already >>> > > > >> > > there >>> > > > >> > > > > in >>> > > > >> > > > > > the secondary storage server. >>> > > > >> > > > > > >>> > > > >> > > > > > Any advice is appreciated. >>> > > > >> > > > > > >>> > > > >> > > > > > Looking forward to your reply, thank you. >>> > > > >> > > > > > >>> > > > >> > > > > > Cheers. >>> > > > >> > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna >>> Santhanam >>> > > > >> <t...@apache.org <mailto:t...@apache.org>> >>> > > > >> >>> > > > >> > > > > wrote: >>> > > > >> > > > > > >>> > > > >> > > > > > > It looks like a previous attempt to start >>> the >>> > > > >> systemVMs has failed >>> > > > >> > > > > > > putting the nfs storage in the avoid set. >>> Did you >>> > > > >> try restarting >>> > > > >> > > your >>> > > > >> > > > > > > management server? >>> > > > >> > > > > > > >>> > > > >> > > > > > > This line leads me to the above mentioned: >>> > > > >> > > > > > > 2013-07-12 13:10:48,236 DEBUG >>> > > > >> > > > > > > >>> > > [storage.allocator.**AbstractStoragePoolAllocator] >>> > > > >> > > (secstorage-1:null) >>> > > > >> > > > > > > StoragePool is in avoid set, skipping this >>> pool >>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, >>> Indra >>> > > > >> Pramana wrote: >>> > > > >> > > > > > > > Dear Wido and all, >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > I have managed to get the hosts, primary >>> and >>> > > > >> secondary storage >>> > > > >> > > > > running. >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > - 2 KVM hypervisor hosts >>> > > > >> > > > > > > > - One RBD primary storage >>> > > > >> > > > > > > > - One NFS primary storage (for system >>> VMs, >>> > > since >>> > > > >> I understand >>> > > > >> > > that >>> > > > >> > > > > system >>> > > > >> > > > > > > > VMs cannot use RBD) >>> > > > >> > > > > > > > - One NFS secondary storage >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > However, now I am having problem with the >>> > > system >>> > > > >> VMs: CPVM and >>> > > > >> > > SSVM, >>> > > > >> > > > > > > unable >>> > > > >> > > > > > > > to start. >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > Excerpt from management-server.log file >>> is >>> > > here: >>> > > > >> > > > > > > > http://pastebin.com/ENkpCALY >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > It seems that the VMs were not able to be >>> > > created >>> > > > >> because unable >>> > > > >> > > to >>> > > > >> > > > > find >>> > > > >> > > > > > > > suitable StoragePools. >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > I understand that system VMs will be >>> using the >>> > > > >> NFS primary >>> > > > >> > > storage >>> > > > >> > > > > > > instead >>> > > > >> > > > > > > > of RBD, so I have confirmed that I am >>> able to >>> > > > >> mount the primary >>> > > > >> > > > > storage >>> > > > >> > > > > > > via >>> > > > >> > > > > > > > NFS and have read and write access, from >>> both >>> > > the >>> > > > >> hypervisor and >>> > > > >> > > the >>> > > > >> > > > > > > > management server. >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > Any advise how can I resolve the problem >>> to >>> > > make >>> > > > >> both the system >>> > > > >> > > VMs >>> > > > >> > > > > > > > created and started? >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > Looking forward to your reply, thank you. >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > Cheers. >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra >>> Pramana >>> > > > >> <in...@sg.or.id <mailto:in...@sg.or.id>> >>> > > > >> >>> > > > >> > > > > wrote: >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > > Hi Wido, >>> > > > >> > > > > > > > > >>> > > > >> > > > > > > > > Thanks for the advice, I'm now able to >>> add >>> > > the >>> > > > >> RBD pool as >>> > > > >> > > primary >>> > > > >> > > > > > > storage. >>> > > > >> > > > > > > > > >>> > > > >> > > > > > > > > Many thanks! :) >>> > > > >> > > > > > > > > >>> > > > >> > > > > > > > > Cheers. >>> > > > >> > > > > > > > > >>> > > > >> > > > > > > > > >>> > > > >> > > > > > > > > On Thursday, July 11, 2013, Wido den >>> > > Hollander >>> > > > >> wrote: >>> > > > >> > > > > > > > > >>> > > > >> > > > > > > > >> Hi, >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers >>> wrote: >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >>> Cc'ing Wido, our resident Ceph >>> expert. ;-) >>> > > > >> > > > > > > > >>> >>> > > > >> > > > > > > > >>> >>> > > > >> > > > > > > > >> Hehe ;) >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM >>> +0800, >>> > > > >> Indra Pramana >>> > > > >> > > wrote: >>> > > > >> > > > > > > > >>> >>> > > > >> > > > > > > > >>>> Dear all, >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> I am installing CloudStack 4.1.0 >>> (upgraded >>> > > > >> from 4.0.2) and I >>> > > > >> > > > > also >>> > > > >> > > > > > > have a >>> > > > >> > > > > > > > >>>> Ceph cluster running. However, I am >>> having >>> > > > >> issues in adding >>> > > > >> > > the >>> > > > >> > > > > RBD >>> > > > >> > > > > > > as >>> > > > >> > > > > > > > >>>> primary storage. Tried to follow the >>> > > > >> instruction here, but >>> > > > >> > > > > unable to >>> > > > >> > > > > > > > >>>> make >>> > > > >> > > > > > > > >>>> it work: >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> >>> > > > >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/< >>> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/> >>> > > > >> < >>> > > > >> > > > > > > >>> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/< >>> > > http://ceph.com/docs/master/rbd/rbd-cloudstack/> >>> > > > >> > >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> I have setup a pool on the Ceph >>> cluster. >>> > > The >>> > > > >> status of the >>> > > > >> > > > > cluster >>> > > > >> > > > > > > is >>> > > > >> > > > > > > > >>>> healthy. Since I am using Ubuntu >>> 12.04.2 >>> > > LTS >>> > > > >> (Precise) for >>> > > > >> > > the >>> > > > >> > > > > > > > >>>> hypervisors, >>> > > > >> > > > > > > > >>>> I also have compiled libvirt >>> manually to >>> > > > >> ensure that the >>> > > > >> > > version >>> > > > >> > > > > > > 0.9.13 >>> > > > >> > > > > > > > >>>> is >>> > > > >> > > > > > > > >>>> installed (previously it's 0.9.8). >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >> You can also use the Ubuntu Cloud >>> Archive, I >>> > > > >> still need to >>> > > > >> > > get the >>> > > > >> > > > > > > docs >>> > > > >> > > > > > > > >> updated for that. >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> I described the process in a blogpost: >>> > > > >> > > > > > > http://blog.widodh.nl/2013/06/****< >>> > > http://blog.widodh.nl/2013/06/**> >>> > > > >> > > > > > > > >> >>> > > > >> > > >>> > > > >> >>> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-** >>> > > > >> **12-04/< >>> > > > >> > > > > > > >>> > > > >> > > > > >>> > > > >> > > >>> > > > >> >>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-** >>> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/< >>> > > >>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/ >>> > > > >>> > > > >> > > > > > > > >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph >>> > > > >> > > > > > > > >>>> ceph> health >>> > > > >> > > > > > > > >>>> HEALTH_OK >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools >>> > > > >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1, >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> root@hv-kvm-01:/home/indra# >>> libvirtd >>> > > > >> --version >>> > > > >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13 >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> I tried to add Primary Storage into >>> the >>> > > > >> Cloudstack zone >>> > > > >> > > which I >>> > > > >> > > > > have >>> > > > >> > > > > > > > >>>> created: >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> Add Primary Storage: >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>>> Zone: my zone name >>> > > > >> > > > > > > > >>>> Pod: my pod name >>> > > > >> > > > > > > > >>>> Cluster: my cluster name >>> > > > >> > > > > > > > >>>> Name: ceph-rbd-pri-storage >>> > > > >> > > > > > > > >>>> Protocol: RBD >>> > > > >> > > > > > > > >>>> RADOS Monitor: my first Ceph >>> monitor IP >>> > > > >> address >>> > > > >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on >>> Ceph >>> > > > >> cluster) >>> > > > >> > > > > > > > >>>> RADOS User: client.admin >>> > > > >> > > > > > > > >>>> RADOS Secret: >>> > > > >> /etc/ceph/ceph.client.admin.****keyring (keyring >>> > > > >> > > > > file >>> > > > >> > > > > > > > >>>> location) >>> > > > >> > > > > > > > >>>> >>> > > > >> > > > > > > > >>> >>> > > > >> > > > > > > > >> This is your problem. That shouldn't >>> be the >>> > > > >> location of the >>> > > > >> > > file, >>> > > > >> > > > > but >>> > > > >> > > > > > > it >>> > > > >> > > > > > > > >> should be the secret, which is a >>> base64 >>> > > > >> encoded string. >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> $ ceph auth list >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> That should tell you what the secret >>> is. >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> Storage Tags: rbd >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> This is the error message when I >>> tried to >>> > > add >>> > > > >> the primary >>> > > > >> > > storage >>> > > > >> > > > > by >>> > > > >> > > > > > > > >> clicking OK: >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> DB Exception on: >>> > > > >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56 >>> > > > >> > > > > **: >>> > > > >> > > > > > > > >> INSERT INTO >>> > > > >> > > > > > > > >> storage_pool (storage_pool.id >>> > > > >> <http://storage_pool.id>, storage_pool.name >>> > > > >> <http://storage_pool.name>, >>> > > > >> >>> > > > >> > > > > storage_pool.uuid, >>> > > > >> > > > > > > > >> storage_pool.pool_type, >>> > > storage_pool.created, >>> > > > >> > > > > > > storage_pool.update_time, >>> > > > >> > > > > > > > >> storage_pool.data_center_id, >>> > > > >> storage_pool.pod_id, >>> > > > >> > > > > > > > >> storage_pool.available_bytes, >>> > > > >> storage_pool.capacity_bytes, >>> > > > >> > > > > > > > >> storage_pool.status, >>> storage_pool.scope, >>> > > > >> > > > > > > storage_pool.storage_provider_**** >>> > > > >> > > > > > > > >> id, >>> > > > >> > > > > > > > >> storage_pool.host_address, >>> > > storage_pool.path, >>> > > > >> > > storage_pool.port, >>> > > > >> > > > > > > > >> storage_pool.user_info, >>> > > > >> storage_pool.cluster_id) VALUES (217, >>> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage', >>> > > > >> > > > > > > > >> >>> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', >>> 'RBD', >>> > > > >> > > > > '2013-07-10 >>> > > > >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', >>> null, >>> > > null, >>> > > > >> null, >>> > > > >> > > _binary'ceph/ >>> > > > >> > > > > > > > >> ceph.client.admin.keyring@10. >>> > > ****237.11.2/sc1< >>> > > > >> > > > > > > http://ceph.client.admin.** >>> > > keyring@10.237.11.2/sc1< >>> http://ceph.client.admin.keyring@10.237.11.2/sc1> >>> > > > >> >', >>> > > > >> > > > > > > > >> 6789, null, 2) >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> On the management-server.log file: >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG >>> > > > >> [cloud.api.ApiServlet] >>> > > > >> > > > > > > > >> (catalina-exec-2:null) >>> > > > >> > > > > > > > >> ===START=== 192.168.0.100 -- GET >>> > > > >> > > > > > > > >> >>> > > > >> >>> > > command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7- >>> > > > >> > > > > > > > >> >>> > > > >> >>> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&** >>> > > > >> > > > > > > > >> >>> > > > >> >>> > > clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-**** >>> > > > >> > > > > > > > >> >>> > > > >> >>> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc >>> > > > >> > > > > > > > >> >>> > > > >> >>> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&** >>> > > > >> > > > > > > > >> >>> > > > >> > > > > >>> > > > >> tags=rbd&response=json&****sessionkey=**** >>> > > > >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%**** >>> > > > >> > > > > > > > >> 3D&_=1373447307839 >>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG >>> > > > >> > > [cloud.storage.****StorageManagerImpl] >>> > > > >> > > > > > > > >> (catalina-exec-2:null) createPool >>> Params @ >>> > > > >> scheme - rbd >>> > > > >> > > > > storageHost - >>> > > > >> > > > > > > null >>> > > > >> > > > > > > > >> hostPath - /ceph/ceph.client >>> > > > >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1 >>> > > > >> <http://admin.keyring@10.237.**11.2/sc1< >>> > > http://admin.keyring@10.237.11.2/sc1>> >>> > > > >> port - -1 >>> > > > >> >>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG >>> > > > >> > > [cloud.storage.****StorageManagerImpl] >>> > > > >> > > > > > > > >> (catalina-exec-2:null) In createPool >>> Setting >>> > > > >> poolId - 217 >>> > > > >> > > uuid - >>> > > > >> > > > > > > > >> >>> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z >>> > > > >> > > > > > > > >> oneId - 2 podId - 2 poolName - >>> > > > >> ceph-rbd-pri-storage >>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG >>> > > > >> [db.Transaction.Transaction] >>> > > > >> > > > > > > > >> (catalina-exec-2:null) Rolling back >>> the >>> > > > >> transaction: Time = 3 >>> > > > >> > > > > Name = >>> > > > >> > > > > > > > >> persist; called by >>> -Transaction.rollbac >>> > > > >> > > > > > > > >> >>> > > > >> >>> k:890-Transaction.removeUpTo:****833-Transaction.close:657-** >>> > > > >> > > > > > > > >> >>> > > > >> TransactionContextBuilder.****interceptException:63-** >>> > > > >> > > > > > > > >> >>> > > > >> > > > > >>> > > > >> >>> > > ComponentInstantiationPostProc****essor$InterceptorDispatcher.** >>> > > > >> **interce >>> > > > >> > > > > > > > >> pt:133-StorageManagerImpl.**** >>> > > > >> createPool:1378-** >>> > > > >> > > > > > > > >> >>> > > > >> >>> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.** >>> > > > >> > > > > > > > >> >>> > > > >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.** >>> > > > >> > > > > > > > >> queueCommand:505-ApiSe >>> > > > >> > > > > > > > >> >>> > > > >> rver.handleRequest:355-****ApiServlet.processRequest:302 >>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR >>> > > > >> [cloud.api.ApiServer] >>> > > > >> > > > > > > (catalina-exec-2:null) >>> > > > >> > > > > > > > >> unhandled exception executing api >>> command: >>> > > > >> createStoragePool >>> > > > >> > > > > > > > >> >>> > > > >> com.cloud.utils.exception.****CloudRuntimeException: DB >>> > > > >> > > Exception >>> > > > >> > > > > on: >>> > > > >> > > > > > > > >> >>> > > > >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****: >>> INSERT >>> > > > >> > > INTO >>> > > > >> > > > > > > > >> storage_pool ( >>> > > > >> > > > > > > > >> storage_pool.id < >>> http://storage_pool.id>, >>> > > > >> >>> > > > >> storage_pool >>> > > > >> > > > > > > > >> .name, storage_pool.uuid, >>> > > > >> storage_pool.pool_type, >>> > > > >> > > > > > > storage_pool.created, >>> > > > >> > > > > > > > >> storage_pool.update_time, >>> > > > >> storage_pool.data_center_id, >>> > > > >> > > > > > > > >> storage_pool.pod_id, >>> > > > >> > > > > > > > >> storage_pool.availab >>> > > > >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes, >>> > > > >> storage_pool.status, >>> > > > >> > > > > > > > >> storage_pool.scope, >>> > > > >> storage_pool.storage_provider_****id, >>> > > > >> > > > > > > > >> storage_pool.host_address, >>> > > storage_pool.path, >>> > > > >> storage_ >>> > > > >> > > > > > > > >> pool.port, storage_pool.user_info, >>> > > > >> storage_pool.cluster_id) >>> > > > >> > > VALUES >>> > > > >> > > > > > > (217, >>> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage', >>> > > > >> > > > > > > > >> >>> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', >>> 'RBD', >>> > > > >> > > '2013-07-1 >>> > > > >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', >>> null, >>> > > > >> null, null, >>> > > > >> > > > > _binary'ceph/ >>> > > > >> > > > > > > > >> ceph.client.admin.keyring@10. >>> > > ****237.11.2/sc1< >>> > > > >> > > > > > > http://ceph.client.admin.** >>> > > keyring@10.237.11.2/sc1< >>> http://ceph.client.admin.keyring@10.237.11.2/sc1> >>> > > > >> >', >>> > > > >> > > > > > > > >> 6789, null, 2) >>> > > > >> > > > > > > > >> at >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > >>> > > > >> > > >>> > > > >> com.cloud.utils.db.****GenericDaoBase.persist(**** >>> > > > >> GenericDaoBase.java:1342) >>> > > > >> > > > > > > > >> at >>> > > > >> > > > > > > > >> >>> > > > >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(** >>> > > > >> > > > > > > > >> StoragePoolDaoImpl.java:232) >>> > > > >> > > > > > > > >> at >>> > > > >> > > > > > > > >> >>> > > > >> >>> > > com.cloud.utils.component.****ComponentInstantiationPostProc**** >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > >>> > > > >> > > > > >>> > > > >> > > >>> > > > >> essor$InterceptorDispatcher.****intercept(**** >>> > > > >> ComponentInstantiationPostProc*** >>> > > > >> > > > > > > > >> *es >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > >>> > > > >> > > > > > > -- >>> > > > >> > > > > > > Prasanna., >>> > > > >> > > > > > > >>> > > > >> > > > > > > ------------------------ >>> > > > >> > > > > > > Powered by BigRock.com >>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > >>> > > > >> > > > > -- >>> > > > >> > > > > Prasanna., >>> > > > >> > > > > >>> > > > >> > > > > ------------------------ >>> > > > >> > > > > Powered by BigRock.com >>> > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > > >>> > > > >> > > -- >>> > > > >> > > Prasanna., >>> > > > >> > > >>> > > > >> > > ------------------------ >>> > > > >> > > Powered by BigRock.com >>> > > > >> > > >>> > > > >> > > >>> > > > >> >>> > > > >> -- >>> > > > >> Prasanna., >>> > > > >> >>> > > > >> ------------------------ >>> > > > >> Powered by BigRock.com >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > > >>> > > >>> > > -- >>> > > Prasanna., >>> > > >>> > > ------------------------ >>> > > Powered by BigRock.com >>> > > >>> > > >>> >>> -- >>> Prasanna., >>> >>> ------------------------ >>> Powered by BigRock.com >>> >>> >> >
