FYI This is now in master and ready for OpenNebula 4.10.Thanks Alvaro for the contribution :)
On Thu, Jul 17, 2014 at 2:11 PM, Alvaro Simon Garcia < alvaro.simongar...@ugent.be> wrote: > Hi Jaime > > Sorry for the late reply, I didn't see your mail before because I was > included in cc. With this patch you don't need to include a new user into > your ceph keyring. If you want to use a ceph datastore you only need to > include the user and pool into the datastore template that's all. In our > case we use livbirt datastore and we have created two ceph datastore for > testing porposes (each one use a different pool as well): > > > $ onedatastore show ceph > DATASTORE 103 > INFORMATION > ID : 103 > NAME : ceph > USER : oneadmin > GROUP : oneadmin > CLUSTER : - > TYPE : IMAGE > DS_MAD : ceph > TM_MAD : ceph > BASE PATH : /var/lib/one//datastores/103 > DISK_TYPE : RBD > > DATASTORE > CAPACITY > TOTAL: : 87.6T > FREE: : 59.2T > USED: : 28.4T > LIMIT: : - > > PERMISSIONS > > OWNER : um- > GROUP : u-- > OTHER : --- > > DATASTORE > TEMPLATE > BASE_PATH="/var/lib/one//datastores/" > BRIDGE_LIST="hyp004.cubone.os" > CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os" > CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > CEPH_USER="libvirt" > CLONE_TARGET="SELF" > DATASTORE_CAPACITY_CHECK="yes" > DISK_TYPE="RBD" > DS_MAD="ceph" > LN_TARGET="NONE" > POOL_NAME="one" > TM_MAD="ceph" > > IMAGES > 7 > 8 > 31 > 32 > 34 > 35 > > > and the second one: > > $ onedatastore show "ceph two" > DATASTORE 104 > INFORMATION > ID : 104 > NAME : ceph two > USER : oneadmin > GROUP : oneadmin > CLUSTER : - > TYPE : IMAGE > DS_MAD : ceph > TM_MAD : ceph > BASE PATH : /var/lib/one//datastores/104 > DISK_TYPE : RBD > > DATASTORE > CAPACITY > TOTAL: : 87.6T > FREE: : 59.2T > USED: : 28.4T > LIMIT: : - > > PERMISSIONS > > OWNER : um- > GROUP : u-- > OTHER : --- > > DATASTORE > TEMPLATE > BASE_PATH="/var/lib/one//datastores/" > BRIDGE_LIST="hyp004.cubone.os" > CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os" > CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > CEPH_USER="libvirt" > CLONE_TARGET="SELF" > DATASTORE_CAPACITY_CHECK="yes" > DISK_TYPE="RBD" > DS_MAD="ceph" > LN_TARGET="NONE" > POOL_NAME="two" > TM_MAD="ceph" > > As you can see we are using different pools in each one (so we don't need > to include the pool name into ceph.conf either, and we are able to use > several ceph clusters) and this change simplifies the ONE conf and ceph > cluster admin as well. > > Cheers > Alvaro > > On 03/07/14 17:08, Jaime Melis wrote: > > Hi Alvaro, > > thanks a lot for the contribution. I haven't tested it yet, but the code > seems to be perfect. > > My main issue here is that I'm not entirely sure if we need this. > Doesn't it make sense for the oneadmin user to be in the keyring and be > able to run rados? You mentioned this, but I'm having some trouble > understanding what do you mean: > > > This feature could be useful as well if you want to monitor several > datastores that are using different ceph pools and users ids. > > You only have to include the id and pool info into the ONE datastore > > template > and the monitoring script will use one or another depending on the DS > conf. > > With the current system you can monitor multiple ceph pools as long as > the oneadmin user has right, isn't that so? > > Joel, would you like to weigh in? would you find this useful? > > cheers, > Jaime > > > On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan <jfon...@opennebula.org> > wrote: > >> Great, thanks! Our ceph dev is traveling right now, he will review the >> patches when he arrives. >> >> Cheers >> >> On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia >> <alvaro.simongar...@ugent.be> wrote: >> > Hi Javier >> > >> > We have modified the ceph monitor to take into account the datastore >> ceph >> > pool and also the ceph user. This is a generic solution that could be >> useful >> > for other datacenters as well, we have created a pull request in github >> if >> > you are agree about this change and you want to include it in the next >> > release. >> > >> > https://github.com/OpenNebula/one/pull/27 >> > >> > We only have modified these lines in >> > /var/lib/one/remotes/datastore/ceph/monitor: >> > >> >> --- monitor.orig.190614 2014-06-19 14:35:24.022755989 +0200 >> >> +++ monitor 2014-06-19 14:49:34.043187892 +0200 >> >> @@ -46,10 +46,12 @@ >> >> while IFS= read -r -d '' element; do >> >> XPATH_ELEMENTS[i++]="$element" >> >> done < <($XPATH /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST >> \ >> >> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME) >> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \ >> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER) >> >> >> >> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}" >> >> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}" >> >> +CEPH_USER="${XPATH_ELEMENTS[j++]}" >> >> >> >> HOST=`get_destination_host` >> >> >> >> @@ -61,7 +63,7 @@ >> >> # ------------ Compute datastore usage ------------- >> >> >> >> MONITOR_SCRIPT=$(cat <<EOF >> >> -$RADOS df | $AWK '{ >> >> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{ >> >> if (\$1 == "total") { >> >> >> >> space = int(\$3/1024) >> > >> > >> > CEPH_USER and POOL_NAME should be mandatory to create the ceph >> datastore. >> > >> > Cheers >> > Alvaro >> > >> > >> > >> >> Hola Javi >> >> >> >> Thanks a lot for your feedback. Yes we will modify the current >> >> monitoring scripts to take into account this. This feature could be >> >> useful as well if you want to monitor several datastores that are using >> >> different ceph pools and users ids. You only have to include the id and >> >> pool info into the ONE datastore template and the monitoring script >> will >> >> use one or another depending on the DS conf. >> >> >> >> >> >> Cheers and thanks! >> >> Alvaro >> >> On 2014-06-17 14:55, Javier Fontan wrote: >> >>> >> >>> CEPH_USER is used when generating the libvirt/kvm deployment file but >> >>> not for DS monitoring: >> >>> >> >>> * Deployment file generation: >> >>> >> >>> >> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461 >> >>> * Monitoring: >> >>> >> https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64 >> >>> >> >>> Ceph is not may area of expertise but you may need to add those >> >>> parameters to the monitor script a maybe to other scripts that use the >> >>> "rados" command. It may also be possible to modify the RADOS command >> >>> to have those parameters instead of modifying all the scripts: >> >>> >> >>> >> >>> >> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40 >> >>> >> >>> As I said I don't know much about Ceph and it may be those credentials >> >>> could be set in a config file or so. >> >>> >> >>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia >> >>> <alvaro.simongar...@ugent.be> wrote: >> >>>> >> >>>> Hi >> >>>> >> >>>> We have included the admin keyring instead of libvirt user and it >> >>>> works... >> >>>> that means that we can run rbd or qemu-img wihtout the libvirt id, >> but >> >>>> is >> >>>> not the best solution. We have included the user into datastore conf: >> >>>> >> >>>> CEPH_USER="libvirt" >> >>>> >> >>>> but it seems that is not used by opennebula at the end >> >>>> >> >>>> Cheers >> >>>> Alvaro >> >>>> >> >>>> >> >>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote: >> >>>>> >> >>>>> Hi all >> >>>>> >> >>>>> >> >>>>> We have included our ONE nodes into Ceph cluster, cephx auth is >> working >> >>>>> but OpenNebula is not able to detect the free space: >> >>>>> >> >>>>> >> >>>>> >> >>>>>> $ onedatastore show 103 >> >>>>>> DATASTORE 103 INFORMATION >> >>>>>> ID : 103 >> >>>>>> NAME : ceph >> >>>>>> USER : oneadmin >> >>>>>> GROUP : oneadmin >> >>>>>> CLUSTER : - >> >>>>>> TYPE : IMAGE >> >>>>>> DS_MAD : ceph >> >>>>>> TM_MAD : ceph >> >>>>>> BASE PATH : /var/lib/one//datastores/103 >> >>>>>> DISK_TYPE : RBD >> >>>>>> >> >>>>>> DATASTORE CAPACITY >> >>>>>> TOTAL: : 0M >> >>>>>> FREE: : 0M >> >>>>>> USED: : 0M >> >>>>>> LIMIT: : - >> >>>>>> >> >>>>>> PERMISSIONS >> >>>>>> OWNER : um- >> >>>>>> GROUP : u-- >> >>>>>> OTHER : --- >> >>>>> >> >>>>> >> >>>>>> $ onedatastore list >> >>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS >> >>>>>> TM >> >>>>>> 0 system 114.8G 85% - 0 sys - >> >>>>>> shared >> >>>>>> 1 default 114.9G 84% - 2 img fs >> >>>>>> ssh >> >>>>>> 2 files 114.9G 84% - 0 fil fs >> >>>>>> ssh >> >>>>>> 103 ceph 0M - - 0 img ceph >> >>>>>> ceph >> >>>>> >> >>>>> >> >>>>> but if we run rados as oneadmin user: >> >>>>> >> >>>>>> $ rados df -p one --id libvirt >> >>>>>> pool name category KB objects clones >> >>>>>> degraded unfound rd rd KB wr >> wr >> >>>>>> KB >> >>>>>> one - 0 0 0 >> >>>>>> 0 0 0 0 0 0 >> >>>>>> total used 1581852 37 >> >>>>>> total avail 140846865180 >> >>>>>> total space 140848447032 >> >>>>> >> >>>>> >> >>>>> It's working correctly (we are using one pool and libvirt ceph id) >> >>>>> >> >>>>> the oned.log only shows this info: >> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore default (1) >> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore files (2) >> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore ceph (103) >> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default (1) >> successfully >> >>>>> monitored. >> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2) successfully >> >>>>> monitored. >> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103) successfully >> >>>>> monitored. >> >>>>> >> >>>>> Any clue about how to debug this issue? >> >>>>> >> >>>>> Thanks in advance! >> >>>>> Alvaro >> >>>>> >> >>>>> >> >>>>> >> >>>>> _______________________________________________ >> >>>>> Users mailing list >> >>>>> Users@lists.opennebula.org >> >>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org >> >>>> >> >>>> _______________________________________________ >> >>>> Users mailing list >> >>>> Users@lists.opennebula.org >> >>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org >> >>> >> >>> >> >> _______________________________________________ >> >> Users mailing list >> >> Users@lists.opennebula.org >> >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org >> > >> > >> >> >> >> -- >> Javier Fontán Muiños >> Developer >> OpenNebula - Flexible Enterprise Cloud Made Simple >> www.OpenNebula.org | @OpenNebula | github.com/jfontan >> _______________________________________________ >> Users mailing list >> Users@lists.opennebula.org >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org >> > > > > -- > Jaime Melis > Project Engineer > OpenNebula - Flexible Enterprise Cloud Made Simple > www.OpenNebula.org | jme...@opennebula.org > > > > _______________________________________________ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > > -- -- Ruben S. Montero, PhD Project co-Lead and Chief Architect OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | rsmont...@opennebula.org | @OpenNebula
_______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org