Ceph file system can added as Primary storage by RBD, Thanks.
Now I meet the problem in creating VM to Ceph.
First I create computer offer and storage offer with tag "ceph", and define
Ceph Primary storage with tag "ceph".
Then I create VM use "ceph" computer offer, It display exception in Log. It
seems the storage size out off usage.
Create VM fail
2013-04-18 10:23:27,559 DEBUG
[storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-1:job-24) Is
storage pool shared? true
2013-04-18 10:23:27,561 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-1:job-24) Checking pool 209 for storage, totalSize:
146673582080, *usedBytes: 33780317214998658*, usedPct: 230309.48542985672,
disable threshold: 0.85
2013-04-18 10:23:27,561 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-1:job-24) Insufficient space on pool: 209 since its usage
percentage: 230309.48542985672 has crossed the
pool.storage.capacity.disablethreshold: 0.85
2013-04-18 10:23:27,561 DEBUG
[storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-1:job-24)
FirstFitStoragePoolAllocator returning 0 suitable storage pools
2013-04-18 10:23:27,561 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24) No suitable pools found for volume:
Vol[8|vm=8|ROOT] under cluster: 1
2013-04-18 10:23:27,562 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24) No suitable pools found
2013-04-18 10:23:27,562 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24) No suitable storagePools found under this Cluster: 1
2013-04-18 10:23:27,562 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24) Could not find suitable Deployment Destination for
this VM under any clusters, returning.
2013-04-18 10:23:27,640 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-1:job-24) VM state transitted from :Starting to Stopped with
event: OperationFailedvm's original host id: null new host id: null host id
before state transition: null
2013-04-18 10:23:27,645 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-1:job-24) Destroying vm VM[User|ceph-1] as it failed to create
management-server.7z 2013-04-18 10:02
I check CS Web GUI and Database, the storage Allocated 30723TB, but
actually the total size is 138GB.
[image: Inline image 1]
mysql> select * from storage_pool;
+-----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+-------------------+----------------+--------------+-----------+-----------------+---------------------+---------+-------------+--------+
| id | name | uuid | pool_type
| port | data_center_id | pod_id | cluster_id | available_bytes |
capacity_bytes | host_address | user_info | path | created
| removed | update_time | status |
+-----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+-------------------+----------------+--------------+-----------+-----------------+---------------------+---------+-------------+--------+
| 200 | primary | d9474b9d-afa1-3737-a13b-df333dae295f | NetworkFilesystem
| 2049 | 1 | 1 | 1 | 10301210624 |
61927849984 | 10.0.0.42 | NULL | /export/primary | 2013-04-17
06:25:47 | NULL | NULL | Up |
| 209 | ceph | 18559bbf-cd1a-3fcc-a4d2-e95cd4f2d78c | RBD
| 6789 | 1 | 1 | 1 | 33780317214998658 |
146673582080
| 10.0.0.41 | : | rbd | 2013-04-18 02:02:00 | NULL
| NULL | Up |
+-----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+-------------------+----------------+--------------+-----------+-----------------+---------------------+---------+-------------+--------+
2 rows in set (0.00 sec)
On Thu, Apr 18, 2013 at 3:54 AM, Wido den Hollander <[email protected]> wrote:
> Hi,
>
>
> On 04/17/2013 03:01 PM, Guangjian Liu wrote:
>
>> I still meet the same result.
>>
>> In ubuntu 12.04,
>> 1. I install libvirt-dev as below,
>> apt-get install libvirt-dev
>> 2. rebuild libvirt, see detail build log in attach.
>> root@ubuntu:~/install/libvirt-**0.10.2# ./autogen.sh
>> running CONFIG_SHELL=/bin/bash /bin/bash ./configure --enable-rbd
>> --no-create --no-recursion
>> configure: WARNING: unrecognized options: --enable-rbd
>>
>
> The correct option is "--with-storage-rbd"
>
> But check the output of configure, it should tell you whether RBD was
> enabled or not.
>
> Then verify again if you can create a RBD storage pool manually via
> libvirt.
>
> Wido
>
> .....
>> make
>> make install
>>
>>
>>
>> On Wed, Apr 17, 2013 at 6:25 PM, Wido den Hollander <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi,
>>
>>
>> On 04/17/2013 11:37 AM, Guangjian Liu wrote:
>>
>> Thanks for your mail, you suggest compile libvirt with RBD enable.
>> I already build libvirt-0.10.2.tar.gz as document
>>
>> http://ceph.com/docs/master/__**rbd/libvirt/<http://ceph.com/docs/master/__rbd/libvirt/>
>>
>>
>> <http://ceph.com/docs/master/**rbd/libvirt/<http://ceph.com/docs/master/rbd/libvirt/>>
>> in my SERVER C(Ubuntu
>> 12.04),
>> Shall I build libvirt-0.10.2.tar.gz with RBD enable? use
>> ./configure
>> --enable-rbd instead autogen.sh?
>>
>>
>> Well, you don't have to add --enable-rbd to configure nor
>> autogen.sh, but you have to make sure the development libraries for
>> librbd are installed.
>>
>> On CentOS do this:
>>
>> yum install librbd-devel
>>
>> And retry autogen.sh for libvirt, it should tell you RBD is enabled.
>>
>> Wido
>>
>> cd libvirt
>> ./autogen.sh
>> make
>> sudo make install
>>
>>
>>
>> On Wed, Apr 17, 2013 at 4:37 PM, Wido den Hollander
>> <[email protected] <mailto:[email protected]>> wrote:
>>
>> Hi,
>>
>>
>> On 04/17/2013 01:44 AM, Guangjian Liu wrote:
>>
>> Create rbd primary storage fail in CS 4.0.1
>> Anybody can help about it!
>>
>> Environment:
>> 1. Server A: CS 4.0.1 OS: RHEL 6.2 x86-64
>> 2. Server B: Ceph 0.56.4 OS: RHEL 6.2 x86-64
>> 3. Server C: KVM/Qemu OS: Ubuntu 12.04
>> compile libvirt and Qemu as document
>> root@ubuntu:/usr/local/lib# virsh version
>> Compiled against library: libvirt 0.10.2
>> Using library: libvirt 0.10.2
>> Using API: QEMU 0.10.2
>> Running hypervisor: QEMU 1.0.0
>>
>>
>> Are you sure both libvirt and Qemu are compiled with RBD
>> enabled?
>>
>> On your CentOS system you should make sure librbd-dev is
>> installed during
>> compilation of libvirt and Qemu.
>>
>> The most important part is the RBD storage pool support in
>> libvirt, that
>> should be enabled.
>>
>> In the e-mail you send me directly I saw this:
>>
>> root@ubuntu:~/scripts# virsh pool-define rbd-pool.xml error:
>> Failed to
>> define pool from rbd-pool.xml error: internal error missing
>> backend for
>> pool type 8
>>
>> That suggest RBD storage pool support is not enabled in
>> libvirt.
>>
>> Wido
>>
>>
>> Problem:
>>
>> create primary storage fail with rbd device.
>>
>> Fail log:
>> 2013-04-16 16:27:14,224 DEBUG
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) createPool Params @ scheme - rbd
>> storageHost -
>> 10.0.0.41 hostPath - /cloudstack port - -1
>> 2013-04-16 16:27:14,270 DEBUG
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) In createPool Setting poolId -
>> 218 uuid -
>> 5924a2df-d658-3119-8aba-**__**f90307683206 zoneId - 4
>>
>> podId - 4 poolName -
>> ceph
>> 2013-04-16 16:27:14,318 DEBUG
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) creating pool ceph on host 18
>> 2013-04-16 16:27:14,320 DEBUG [agent.transport.Request]
>> (catalina-exec-9:null) Seq 18-1625162275: Sending { Cmd
>> , MgmtId:
>> 37528005876872, via: 18, Ver: v1, Flags: 100011,
>> [{"CreateStoragePoolCommand":{**
>> __**"add":true,"pool":{"id":**218,__"**
>> uuid":"5924a2df-d658-3119-**__**
>> 8aba-f90307683206","host":"10.**__**
>> 0.0.41","path":"cloudstack","***
>> __*userInfo":":","port":6789,"****
>> type":"RBD"},"localPath":"/**_**
>> _mnt//3cf4f0e8-781d-39d8-b81c-***__*
>>
>>
>> 9896da212335","wait":0}}]
>> }
>> 2013-04-16 16:27:14,323 DEBUG [agent.transport.Request]
>> (AgentManager-Handler-2:null) Seq 18-1625162275:
>> Processing: { Ans: ,
>> MgmtId: 37528005876872, via: 18, Ver: v1, Flags: 10,
>> [{"Answer":{"result":true,"**_**
>> _details":"success","wait":0}}**]
>>
>> }
>>
>> 2013-04-16 16:27:14,323 DEBUG [agent.transport.Request]
>> (catalina-exec-9:null) Seq 18-1625162275: Received: {
>> Ans: , MgmtId:
>> 37528005876872, via: 18, Ver: v1, Flags: 10, { Answer } }
>> 2013-04-16 16:27:14,323 DEBUG
>> [agent.manager.**__**AgentManagerImpl]
>>
>>
>> (catalina-exec-9:null) Details from executing class
>> com.cloud.agent.api.**__**CreateStoragePoolCommand:
>> success
>>
>> 2013-04-16 16:27:14,323 DEBUG
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) In createPool Adding the pool to
>> each of the hosts
>> 2013-04-16 16:27:14,323 DEBUG
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) Adding pool ceph to host 18
>> 2013-04-16 16:27:14,326 DEBUG [agent.transport.Request]
>> (catalina-exec-9:null) Seq 18-1625162276: Sending { Cmd
>> , MgmtId:
>> 37528005876872, via: 18, Ver: v1, Flags: 100011,
>> [{"ModifyStoragePoolCommand":{**
>> __**"add":true,"pool":{"id":**218,__"**
>> uuid":"5924a2df-d658-3119-**__**
>> 8aba-f90307683206","host":"10.**__**
>> 0.0.41","path":"cloudstack","***
>> __*userInfo":":","port":6789,"****
>> type":"RBD"},"localPath":"/**_**
>> _mnt//3cf4f0e8-781d-39d8-b81c-***__*
>>
>>
>> 9896da212335","wait":0}}]
>> }
>> 2013-04-16 16:27:14,411 DEBUG [agent.transport.Request]
>> (AgentManager-Handler-6:null) Seq 18-1625162276:
>> Processing: { Ans: ,
>> MgmtId: 37528005876872, via: 18, Ver: v1, Flags: 10,
>> [{"Answer":{"result":false,"****__details":"java.lang.**
>> NullPointerException\n\tat
>> com.cloud.hypervisor.kvm.**__**
>> storage.LibvirtStorageAdaptor.**__**
>> createStoragePool(**__**LibvirtStorageAdaptor.java:**_**
>> _462)\n\tat
>> com.cloud.hypervisor.kvm.**__**
>> storage.KVMStoragePoolManager.**__**
>> createStoragePool(**__**KVMStoragePoolManager.java:57)**
>> __**\n\tat
>> com.cloud.hypervisor.kvm.**__**resource.**__**
>> LibvirtComputingResource.**__**execute(
>> **LibvirtComputingResource.__**java:**2087)\n\tat
>> com.cloud.hypervisor.kvm.**__**resource.**__**
>> LibvirtComputingResource.**
>> executeRequest(**__**LibvirtComputingResource.java:**
>> __**1053)\n\tat
>> com.cloud.agent.Agent.**__**
>> processRequest(Agent.java:518)**__**\n\tat
>> com.cloud.agent.Agent$**__**AgentRequestHandler.doTask(**
>> Agent.java:831)\n\tat
>> com.cloud.utils.nio.Task.run(***__*Task.java:83)\n\tat
>> java.util.concurrent.**__**ThreadPoolExecutor.runWorker(*
>> **__*
>> ThreadPoolExecutor.java:1146)\**__**n\tat
>> java.util.concurrent.**__**ThreadPoolExecutor$Worker.run(
>> **__**
>> ThreadPoolExecutor.java:615)\***__*n\tat
>> java.lang.Thread.run(Thread.****__java:679)\n","wait":0}}]
>> }
>>
>>
>> 2013-04-16 16:27:14,412 DEBUG [agent.transport.Request]
>> (catalina-exec-9:null) Seq 18-1625162276: Received: {
>> Ans: , MgmtId:
>> 37528005876872, via: 18, Ver: v1, Flags: 10, { Answer } }
>> 2013-04-16 16:27:14,412 DEBUG
>> [agent.manager.**__**AgentManagerImpl]
>>
>>
>> (catalina-exec-9:null) Details from executing class
>> com.cloud.agent.api.**__**ModifyStoragePoolCommand:
>> java.lang.NullPointerException
>> at
>> com.cloud.hypervisor.kvm.**__**
>> storage.LibvirtStorageAdaptor.**__**
>> createStoragePool(**__**LibvirtStorageAdaptor.java:**_**
>> _462)
>> at
>> com.cloud.hypervisor.kvm.**__**
>> storage.KVMStoragePoolManager.**__**
>> createStoragePool(**__**KVMStoragePoolManager.java:57)
>> at
>> com.cloud.hypervisor.kvm.**__**resource.**__**
>> LibvirtComputingResource.**__**execute(
>> **LibvirtComputingResource.__**java:**2087)
>> at
>> com.cloud.hypervisor.kvm.**__**resource.**__**
>> LibvirtComputingResource.**
>> executeRequest(**__**LibvirtComputingResource.java:**
>> __**1053)
>> at
>> com.cloud.agent.Agent.**__**
>> processRequest(Agent.java:518)
>> at
>> com.cloud.agent.Agent$**__**AgentRequestHandler.doTask(**
>> Agent.java:831)
>> at com.cloud.utils.nio.Task.run(***
>> __*Task.java:83)
>> at
>> java.util.concurrent.**__**ThreadPoolExecutor.runWorker(*
>> **__*
>> ThreadPoolExecutor.java:1146)
>> at
>> java.util.concurrent.**__**ThreadPoolExecutor$Worker.run(
>> **__**
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.****__java:679)
>>
>>
>> 2013-04-16 16:27:14,451 WARN
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) Unable to establish a connection
>> between
>> Host[-18-Routing] and Pool[218|RBD]
>> com.cloud.exception.**__**StorageUnavailableException:
>>
>> Resource
>>
>> [StoragePool:218]
>> is unreachable: Unable establish connection from storage
>> head to storage
>> pool 218 due to java.lang.NullPointerException
>> at
>> com.cloud.hypervisor.kvm.**__**
>> storage.LibvirtStorageAdaptor.**__**
>> createStoragePool(**__**LibvirtStorageAdaptor.java:**_**
>> _462)
>> at
>> com.cloud.hypervisor.kvm.**__**
>> storage.KVMStoragePoolManager.**__**
>> createStoragePool(**__**KVMStoragePoolManager.java:57)
>> at
>> com.cloud.hypervisor.kvm.**__**resource.**__**
>> LibvirtComputingResource.**__**execute(
>> **LibvirtComputingResource.__**java:**2087)
>> at
>> com.cloud.hypervisor.kvm.**__**resource.**__**
>> LibvirtComputingResource.**
>> executeRequest(**__**LibvirtComputingResource.java:**
>> __**1053)
>> at
>> com.cloud.agent.Agent.**__**
>> processRequest(Agent.java:518)
>> at
>> com.cloud.agent.Agent$**__**AgentRequestHandler.doTask(**
>> Agent.java:831)
>> at com.cloud.utils.nio.Task.run(***
>> __*Task.java:83)
>> at
>> java.util.concurrent.**__**ThreadPoolExecutor.runWorker(*
>> **__*
>> ThreadPoolExecutor.java:1146)
>> at
>> java.util.concurrent.**__**ThreadPoolExecutor$Worker.run(
>> **__**
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.****__java:679)
>>
>> at
>> com.cloud.storage.**__**StorageManagerImpl.**__**
>> connectHostToSharedPool(**
>> StorageManagerImpl.java:1685)
>> at
>> com.cloud.storage.**__**StorageManagerImpl.createPool(**
>> __**
>> StorageManagerImpl.java:1450)
>> at
>> com.cloud.storage.**__**StorageManagerImpl.createPool(**
>> __**
>> StorageManagerImpl.java:215)
>> at
>> com.cloud.api.commands.**__**
>> CreateStoragePoolCmd.execute(***__*
>> CreateStoragePoolCmd.java:120)
>> at
>> com.cloud.api.ApiDispatcher.****
>> __dispatch(ApiDispatcher.java:****
>> 138)
>> at
>> com.cloud.api.ApiServer.**__**
>> queueCommand(ApiServer.java:****__543)
>> at
>> com.cloud.api.ApiServer.**__**
>> handleRequest(ApiServer.java:***__*422)
>> at
>> com.cloud.api.ApiServlet.**__**
>> processRequest(ApiServlet.**
>> java:304)
>> at
>> com.cloud.api.ApiServlet.**__**doGet(ApiServlet.java:63)
>> at javax.servlet.http.**__**
>> HttpServlet.service(**
>> HttpServlet.java:617)
>> at javax.servlet.http.**__**
>> HttpServlet.service(**
>> HttpServlet.java:717)
>> at
>> org.apache.catalina.core.**__**
>> ApplicationFilterChain.**__**internalDoFilter(**
>> ApplicationFilterChain.java:****__290)
>> at
>> org.apache.catalina.core.**__**
>> ApplicationFilterChain.**__**doFilter(**
>> ApplicationFilterChain.java:****__206)
>> at
>> org.apache.catalina.core.**__**
>> StandardWrapperValve.invoke(**
>> StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.**__**
>> StandardContextValve.invoke(**
>> StandardContextValve.java:191)
>> at
>> org.apache.catalina.core.**__**
>> StandardHostValve.invoke(**
>> StandardHostValve.java:127)
>> at
>> org.apache.catalina.valves.**_**
>> _ErrorReportValve.invoke(**
>> ErrorReportValve.java:102)
>> at
>> org.apache.catalina.valves.**_**_AccessLogValve.invoke(**
>> AccessLogValve.java:555)
>> at
>> org.apache.catalina.core.**__**
>> StandardEngineValve.invoke(**
>> StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.**
>> __**CoyoteAdapter.service(**
>> CoyoteAdapter.java:298)
>> at
>> org.apache.coyote.http11.**__**
>> Http11NioProcessor.process(**
>> Http11NioProcessor.java:889)
>> at
>> org.apache.coyote.http11.**__**Http11NioProtocol$**__**
>> Http11ConnectionHandler.**
>> process(Http11NioProtocol.**__**java:721)
>> at
>> org.apache.tomcat.util.net.**_**
>> _NioEndpoint$SocketProcessor.****
>> run(NioEndpoint.java:2260)
>> at
>> java.util.concurrent.**__**ThreadPoolExecutor.runWorker(*
>> **__*
>> ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.**__**ThreadPoolExecutor$Worker.run(
>> **__**
>> ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.****__java:679)
>>
>> 2013-04-16 16:27:14,452 WARN
>> [cloud.storage.**__**StorageManagerImpl]
>>
>>
>> (catalina-exec-9:null) No host can access storage pool
>> Pool[218|RBD] on
>> cluster 5
>> 2013-04-16 16:27:14,504 WARN [cloud.api.ApiDispatcher]
>> (catalina-exec-9:null) class
>> com.cloud.api.**__**ServerApiException : Failed
>>
>> to
>> add storage pool
>> 2013-04-16 16:27:15,293 DEBUG
>> [agent.manager.**__**AgentManagerImpl]
>>
>>
>> (AgentManager-Handler-12:null) Ping from 18
>> ^C
>> [root@RDR02S02 management]#
>>
>>
>>
>>
>>
>>
>>
>> --
>> Guangjian
>>
>
--
Guangjian
