After I changed the NFS server configuration to allow mount to all
sub-directories, re-export the NFS and voila, the system was able to
download the template and now both the system VMs (CPVM and SSVM) are
running!
Many thanks for your help! :)
Cheers.
On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <in...@sg.or.id
<mailto:in...@sg.or.id>> wrote:
Hi Prasanna,
Good day to you, and thank you for your e-mail.
Yes, the cloudstack-agent service is running on both the KVM hosts.
There is no "cloud" user being created though, when I installed the
agent. I installed the agent as root.
root@hv-kvm-01:/home/indra# service cloudstack-agent status
* cloud-agent is running
root@hv-kvm-01:/home/indra# su - cloud
Unknown id: cloud
Please advise how can I resolve this problem, shall I create the
Unix "cloud" user manually? Basically I follow this instruction to
prepare the KVM host and install the CloudStack agent:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
with this instruction from Wido on how to prepare libvirt with Ceph
RBD storage pool support:
http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
I also have checked /var/log/cloud/agent/agent.log and I don't see
any error messages, except this error message which will show up
every time I restart the agent:
2013-07-12 15:22:47,454 ERROR [cloud.resource.ServerResourceBase]
(main:null) Nics are not configured!
2013-07-12 15:22:47,459 INFO [cloud.resource.ServerResourceBase]
(main:null) Designating private to be nic eth0.5
More logs can be found here: http://pastebin.com/yeNmCt7S
I have configured the network bridges on the NIC interface as per
this instruction:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
On the zone, I used advanced network configuration with just one
physical network for management, public and guest/private. I didn't
include storage, which by default the traffic will use the
management VLAN network.
Please advise if there's anything else I might have been missing.
Looking forward to your reply, thank you.
Cheers.
On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <t...@apache.org
<mailto:t...@apache.org>> wrote:
Indeed, cloudstack will go through the allocation to startup the
system VMs too. So that process is failing to recognize the volume
(.qcow2) present on your NFS storage.
Can you check if your cloudstack agent service is running with
the KVM
host? And it should've created the user cloud. $id cloud to check if
there is a user.
Did you see what's happening in the agent logs? These are under
/var/log/cloud/ on your host when the systemVMs are coming up.
If the
logs are not showing any useful information you can turn on debug
level for more verbosity.
See here: https://cwiki.apache.org/confluence/x/FgPMAQ
On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> Hi Prasanna,
>
> Good day to you, and thank you for your e-mail.
>
> Yes, when I export the NFS, I set the permission so that
normal user will
> be able to have read/write access to the files (no_root_squash).
>
> I have tested and I can have read/write access from my KVM
hosts using
> normal user. BTW, there's no "cloud" user on the hosts, I
believe it's not
> created during cloudstack-agent installation?
>
> In any case, do you think the template issue and the storage pool
> allocation issue might be related, or are they two different
problems
> altogether?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
<t...@apache.org <mailto:t...@apache.org>> wrote:
>
> > Can you access the file as user root? Or user cloud? The
cloudstack
> > agent on your KVM host runs as user cloud and the NFS
permissions
> > might be disallowing the volume (.qcow2) from being accessed.
> >
> > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
> > > Hi Prasanna,
> > >
> > > Good day to you, and thank you for your e-mail.
> > >
> > > Yes, the file exists. I can access the file from the
management server
> > and
> > > the two hypervisors hosts if I mount manually.
> > >
> > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
> > > -la
> > > total
> > > 1418787
> > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > > .
> > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > > ..
> > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > > 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
> > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > > template.properties
> > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
> > > pwd
> > > /mnt/vol1/sec-storage/template/tmpl/1/3
> > >
> > >
> > > Any advise?
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> > >
> > >
> > >
> > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
<t...@apache.org <mailto:t...@apache.org>>
> > wrote:
> > >
> > > > Can you check whether there is a file at:
> > > >
nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > > >
> > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana
wrote:
> > > > > Hi Prasanna,
> > > > >
> > > > > Thanks for your e-mail.
> > > > >
> > > > > I have tried restarting the management server, and
the problem still
> > > > > persists. I even tried to re-do the installation and
configuration
> > again
> > > > > from scratch last night, but the problem still there.
> > > > >
> > > > > I also noted that on the beginning of the logs, I
found some error
> > > > messages
> > > > > saying that the template cannot be downloaded to the
pool. See this
> > logs:
> > > > >
> > > > > http://pastebin.com/BY1AVJ08
> > > > >
> > > > > It says it failed because cannot get volume from the
pool. Could it
> > be
> > > > > related, i.e. the absence of the template caused the
system VMs
> > cannot be
> > > > > created and started?
> > > > >
> > > > > I have ensured that I downloaded the system VM
template using
> > > > > cloud-install-sys-tmplt and verified that the
template is already
> > there
> > > > in
> > > > > the secondary storage server.
> > > > >
> > > > > Any advice is appreciated.
> > > > >
> > > > > Looking forward to your reply, thank you.
> > > > >
> > > > > Cheers.
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
<t...@apache.org <mailto:t...@apache.org>>
> > > > wrote:
> > > > >
> > > > > > It looks like a previous attempt to start the
systemVMs has failed
> > > > > > putting the nfs storage in the avoid set. Did you
try restarting
> > your
> > > > > > management server?
> > > > > >
> > > > > > This line leads me to the above mentioned:
> > > > > > 2013-07-12 13:10:48,236 DEBUG
> > > > > > [storage.allocator.AbstractStoragePoolAllocator]
> > (secstorage-1:null)
> > > > > > StoragePool is in avoid set, skipping this pool
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
Pramana wrote:
> > > > > > > Dear Wido and all,
> > > > > > >
> > > > > > > I have managed to get the hosts, primary and
secondary storage
> > > > running.
> > > > > > >
> > > > > > > - 2 KVM hypervisor hosts
> > > > > > > - One RBD primary storage
> > > > > > > - One NFS primary storage (for system VMs, since
I understand
> > that
> > > > system
> > > > > > > VMs cannot use RBD)
> > > > > > > - One NFS secondary storage
> > > > > > >
> > > > > > > However, now I am having problem with the system
VMs: CPVM and
> > SSVM,
> > > > > > unable
> > > > > > > to start.
> > > > > > >
> > > > > > > Excerpt from management-server.log file is here:
> > > > > > > http://pastebin.com/ENkpCALY
> > > > > > >
> > > > > > > It seems that the VMs were not able to be created
because unable
> > to
> > > > find
> > > > > > > suitable StoragePools.
> > > > > > >
> > > > > > > I understand that system VMs will be using the
NFS primary
> > storage
> > > > > > instead
> > > > > > > of RBD, so I have confirmed that I am able to
mount the primary
> > > > storage
> > > > > > via
> > > > > > > NFS and have read and write access, from both the
hypervisor and
> > the
> > > > > > > management server.
> > > > > > >
> > > > > > > Any advise how can I resolve the problem to make
both the system
> > VMs
> > > > > > > created and started?
> > > > > > >
> > > > > > > Looking forward to your reply, thank you.
> > > > > > >
> > > > > > > Cheers.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
<in...@sg.or.id <mailto:in...@sg.or.id>>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Wido,
> > > > > > > >
> > > > > > > > Thanks for the advice, I'm now able to add the
RBD pool as
> > primary
> > > > > > storage.
> > > > > > > >
> > > > > > > > Many thanks! :)
> > > > > > > >
> > > > > > > > Cheers.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thursday, July 11, 2013, Wido den Hollander
wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > > > > >>
> > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >> Hehe ;)
> > > > > > > >>
> > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
Indra Pramana
> > wrote:
> > > > > > > >>>
> > > > > > > >>>> Dear all,
> > > > > > > >>>>
> > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
from 4.0.2) and I
> > > > also
> > > > > > have a
> > > > > > > >>>> Ceph cluster running. However, I am having
issues in adding
> > the
> > > > RBD
> > > > > > as
> > > > > > > >>>> primary storage. Tried to follow the
instruction here, but
> > > > unable to
> > > > > > > >>>> make
> > > > > > > >>>> it work:
> > > > > > > >>>>
> > > > > > > >>>>
http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > > > > >>>>
> > > > > > > >>>> I have setup a pool on the Ceph cluster. The
status of the
> > > > cluster
> > > > > > is
> > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS
(Precise) for
> > the
> > > > > > > >>>> hypervisors,
> > > > > > > >>>> I also have compiled libvirt manually to
ensure that the
> > version
> > > > > > 0.9.13
> > > > > > > >>>> is
> > > > > > > >>>> installed (previously it's 0.9.8).
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >> You can also use the Ubuntu Cloud Archive, I
still need to
> > get the
> > > > > > docs
> > > > > > > >> updated for that.
> > > > > > > >>
> > > > > > > >> I described the process in a blogpost:
> > > > > > http://blog.widodh.nl/2013/06/**
> > > > > > > >>
> >
a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > > > > >
> > > >
> >
http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > > > > >
> > > > > > > >>
> > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > > > > >>>> ceph> health
> > > > > > > >>>> HEALTH_OK
> > > > > > > >>>>
> > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > > > > >>>>
> > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > > > > >>>>
> > > > > > > >>>> I tried to add Primary Storage into the
Cloudstack zone
> > which I
> > > > have
> > > > > > > >>>> created:
> > > > > > > >>>>
> > > > > > > >>>> Add Primary Storage:
> > > > > > > >>>>
> > > > > > > >>>> Zone: my zone name
> > > > > > > >>>> Pod: my pod name
> > > > > > > >>>> Cluster: my cluster name
> > > > > > > >>>> Name: ceph-rbd-pri-storage
> > > > > > > >>>> Protocol: RBD
> > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > > > > >>>> RADOS User: client.admin
> > > > > > > >>>> RADOS Secret:
/etc/ceph/ceph.client.admin.**keyring (keyring
> > > > file
> > > > > > > >>>> location)
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >> This is your problem. That shouldn't be the
location of the
> > file,
> > > > but
> > > > > > it
> > > > > > > >> should be the secret, which is a base64
encoded string.
> > > > > > > >>
> > > > > > > >> $ ceph auth list
> > > > > > > >>
> > > > > > > >> That should tell you what the secret is.
> > > > > > > >>
> > > > > > > >> Storage Tags: rbd
> > > > > > > >>
> > > > > > > >> This is the error message when I tried to add
the primary
> > storage
> > > > by
> > > > > > > >> clicking OK:
> > > > > > > >>
> > > > > > > >> DB Exception on:
> > com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> > > > **:
> > > > > > > >> INSERT INTO
> > > > > > > >> storage_pool (storage_pool.id
<http://storage_pool.id>, storage_pool.name
<http://storage_pool.name>,
> > > > storage_pool.uuid,
> > > > > > > >> storage_pool.pool_type, storage_pool.created,
> > > > > > storage_pool.update_time,
> > > > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > > > > >> storage_pool.available_bytes,
storage_pool.capacity_bytes,
> > > > > > > >> storage_pool.status, storage_pool.scope,
> > > > > > storage_pool.storage_provider_**
> > > > > > > >> id,
> > > > > > > >> storage_pool.host_address, storage_pool.path,
> > storage_pool.port,
> > > > > > > >> storage_pool.user_info,
storage_pool.cluster_id) VALUES (217,
> > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > >>
_binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > > '2013-07-10
> > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null,
null,
> > _binary'ceph/
> > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > >> 6789, null, 2)
> > > > > > > >>
> > > > > > > >> On the management-server.log file:
> > > > > > > >>
> > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
[cloud.api.ApiServlet]
> > > > > > > >> (catalina-exec-2:null)
> > > > > > > >> ===START=== 192.168.0.100 -- GET
> > > > > > > >>
command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > > > > >>
74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > > > > >>
clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > > > > >>
rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > > > > >>
eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > > > > >>
> > > >
tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > > > > >> 3D&_=1373447307839
> > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > [cloud.storage.**StorageManagerImpl]
> > > > > > > >> (catalina-exec-2:null) createPool Params @
scheme - rbd
> > > > storageHost -
> > > > > > null
> > > > > > > >> hostPath - /ceph/ceph.client
> > > > > > > >> .admin.keyring@10.237.11.2/sc1
<http://admin.keyring@10.237.11.2/sc1> port - -1
> > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > [cloud.storage.**StorageManagerImpl]
> > > > > > > >> (catalina-exec-2:null) In createPool Setting
poolId - 217
> > uuid -
> > > > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > > > > >> oneId - 2 podId - 2 poolName -
ceph-rbd-pri-storage
> > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
[db.Transaction.Transaction]
> > > > > > > >> (catalina-exec-2:null) Rolling back the
transaction: Time = 3
> > > > Name =
> > > > > > > >> persist; called by -Transaction.rollbac
> > > > > > > >>
k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > > > > >>
TransactionContextBuilder.**interceptException:63-**
> > > > > > > >>
> > > >
ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > > > > >>
StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > > > > >>
execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > > > > >> queueCommand:505-ApiSe
> > > > > > > >>
rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > > > > >> 2013-07-10 17:08:28,923 ERROR
[cloud.api.ApiServer]
> > > > > > (catalina-exec-2:null)
> > > > > > > >> unhandled exception executing api command:
createStoragePool
> > > > > > > >>
com.cloud.utils.exception.**CloudRuntimeException: DB
> > Exception
> > > > on:
> > > > > > > >>
com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT
> > INTO
> > > > > > > >> storage_pool (
> > > > > > > >> storage_pool.id <http://storage_pool.id>,
storage_pool
> > > > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > > > > storage_pool.created,
> > > > > > > >> storage_pool.update_time,
storage_pool.data_center_id,
> > > > > > > >> storage_pool.pod_id,
> > > > > > > >> storage_pool.availab
> > > > > > > >> le_bytes, storage_pool.capacity_bytes,
storage_pool.status,
> > > > > > > >> storage_pool.scope,
storage_pool.storage_provider_**id,
> > > > > > > >> storage_pool.host_address, storage_pool.path,
storage_
> > > > > > > >> pool.port, storage_pool.user_info,
storage_pool.cluster_id)
> > VALUES
> > > > > > (217,
> > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > >>
_binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > '2013-07-1
> > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
null, null,
> > > > _binary'ceph/
> > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > >> 6789, null, 2)
> > > > > > > >> at
> > > > > > > >>
> > > > > >
> >
com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > > > > >> at
> > > > > > > >>
com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > > > > >> StoragePoolDaoImpl.java:232)
> > > > > > > >> at
> > > > > > > >>
com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > > > > >>
> > > > > >
> > > >
> >
essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > > > > >> *es
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > > > --
> > > > > > Prasanna.,
> > > > > >
> > > > > > ------------------------
> > > > > > Powered by BigRock.com
> > > > > >
> > > > > >
> > > >
> > > > --
> > > > Prasanna.,
> > > >
> > > > ------------------------
> > > > Powered by BigRock.com
> > > >
> > > >
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
