I know this is WAY delayed, but regarding your Ceph question: OpenNebula is capable of doing all of your RBD image management (creation/deletion/etc.), so you shouldn't need to manually create any images on the back end, and you definitely do not need to use CephFS (not really ready for production use yet anyway).
In our particular setup we do the following: * We change the 'system' datastore to use the 'SSH' transfer manager, so we don't have to worry about linking up the OpenNebula system to the Ceph cluster (we use a separate 10Gb network for storage). We have to modify the premigrate/postmigrate scripts for the SSH transfer manager to ensure VM deployment information is copied to each KVM host (else migrations fail). * Have a Ceph cluster created, and copy the ceph.conf and ceph.keyring to the /etc/ceph directory of every node in the cluster (minus the OpenNebula system), and ensure that the oneadmin user you create has at a minimum read permissions on the file (we change ownership to root.oneadmin and apply 640 permissions). I know on the roadmap for 4.2 is a feature for making the integration with Ceph more secure (without relying on the keyring being on each node, and security being defined within each Libvirt deployment file directly). * We have a dedicated system for management of images that OpenNebula connects to (as we've found connecting directly to a monitor sometimes seems to cause issues when dealing with large images, nothing catastrophic but the mon daemon sometimes stops and restarts. Still chasing that one down.) and define that dedicate system in the ceph.conf file in the DS mad. * A oneadmin account is created as per documentation on the OpenNebula system, the KVM Hosts, and the dedicated image management system. All SSH keys are transferred to this dedicated system as well. * The default storage pool is 'one', so you can have a datastore created without defining the pool name, create a pool named 'one' in the Ceph cluster and you should be off to the races. The nice thing about the Ceph implementation in OpenNebula is that you can define a pool name for each datastore created, so that way when you start using larger deployments of OpenNebula/Ceph, you can segment off your VMs to different Datastores/Pools (for instance, if you have an array/pool with dedicated SSDs for extremely fast I/O, you can have that in the same cluster, just defined in a separate Datastore/Pool, with your CRUSH map configured accordingly). Ceph is very extensible. We've been using Ceph with OpenNebula since 3.8.3 with our own driver and it has been working very well for us. The testing of the integration with 4.0 has been going well for us (we made some slight modifications that we submitted to take advantage of RBD Format 2 images for copy-on-write cloning of non-persistent images) and so far have only found a couple of issues (which are already resolved). If you have any additional questions please don't hesitate to ask! ----- Original Message ----- From: "Jaime Melis" <jme...@opennebula.org> To: "Jon" <three1...@gmail.com> Cc: "Users OpenNebula" <users@lists.opennebula.org> Sent: Wednesday, April 10, 2013 6:37:30 AM Subject: Re: [one-users] [Docs] Problem with OpenVSwitch and some questions about Ceph Hi Jon sorry for the delay in the answer. You can always contibute to the community wiki: http://wiki.opennebula.org/ thanks! cheers, Jaime On Sun, Mar 31, 2013 at 10:11 PM, Jon < three1...@gmail.com > wrote: Hello Jamie, Thanks for the clarification, I didn't realize that install_novnc.sh being in two different locations was a bug. I hadn't installed OpenNebula on anything other than a Debian / Ubuntu system, so this makes sense. is there any userspace documentation we can contribute to? Even if not the documentation itself, perhaps a wiki? I didn't see one but that doesn't mean there isn't one. Thanks, Jon A On Thu, Mar 28, 2013 at 2:56 PM, Jaime Melis < jme...@opennebula.org > wrote: 3) Documentation The fact that install_novnc.sh is being installed to two separate locations is a bug that has already been fixed. With regard to the "/usr/share/opennebula" issue, I'm afraid that's because of the Debian/Ubuntu packaging policy. For other distros (CentOS and openSUSE for example) the path is the one that appears in the documentation. To makes things easier for users I think this should be reflected in the README.debian file, and in the platform notes in the documentation. So thanks a lot for pointing this out. I created this feature to follow this problem: http://dev.opennebula.org/issues/1844 cheers, Jaime On Thu, Mar 28, 2013 at 1:06 PM, Jon < three1...@gmail.com > wrote: Hello All, I've just installed OpenNebula 3.9.80 and I have to say this is amazing. Everything works so smoothly. Anyway, down to business. OpenVSwitch: I've installed and configured OpenVSwitch and am able to manually add the OVS config using libvirt, then launch a VM, >> <interface type='bridge'> >> <source bridge='ovsbr0'/> >> <virtualport type='openvswitch'> >> </virtualport> >> <model type='virtio'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> >> </interface> Which creates the device in ovs: >> system@ovsbr0: >> lookups: hit:1346333 missed:46007 lost:0 >> flows: 8 >> port 0: ovsbr0 (internal) >> port 1: eth0 >> port 2: br0 (internal) >> port 7: vnet0 However, when I attempt to create a virtual network without assigning an IP and instantiate the template I get the error: >> [TemplateInstantiate] Error allocating a new virtual machine. Cannot get >> IP/MAC lease from virtual network 0. The template of the virtual network is: >> oneadmin@loki:~$ onevnet show testnet1 >> VIRTUAL NETWORK 0 INFORMATION >> ID : 0 >> NAME : testnet1 >> USER : oneadmin >> GROUP : oneadmin >> CLUSTER : - >> TYPE : FIXED >> BRIDGE : ovsbr0 >> VLAN : No >> USED LEASES : 0 >> >> PERMISSIONS >> OWNER : um- >> GROUP : --- >> OTHER : --- >> >> VIRTUAL NETWORK TEMPLATE >> >> >> VIRTUAL MACHINES >> If I add an IP to the vnet, I get the following template and error logs (full vm log attached, I think I've identified the relevant line): >> Thu Mar 28 10:34:05 2013 [VMM][E]: post: Command "sudo /usr/bin/ovs-ofctl >> add-flow ovsbr0 >> in_port=,dl_src=02:00:44:47:83:43,priority=40000,actions=normal" failed. >> oneadmin@loki:~$ onevnet show testnet1 >> VIRTUAL NETWORK 0 INFORMATION >> ID : 0 >> NAME : testnet1 >> USER : oneadmin >> GROUP : oneadmin >> CLUSTER : - >> TYPE : FIXED >> BRIDGE : ovsbr0 >> VLAN : No >> USED LEASES : 1 >> >> PERMISSIONS >> OWNER : um- >> GROUP : --- >> OTHER : --- >> >> VIRTUAL NETWORK TEMPLATE >> >> >> USED LEASES >> LEASE=[ MAC="02:00:44:47:83:43", IP="192.168.0.2", >> IP6_LINK="fe80::400:44ff:fe47:8343", USED="1", VID="7" ] >> >> VIRTUAL MACHINES >> >> ID USER GROUP NAME STAT UCPU UMEM HOST TIME >> 7 oneadmin oneadmin template-4-7 fail 0 0K 0d 00h00 >> root@loki:~# cat /var/log/openvswitch/ovs-vswitchd.log >> Mar 28 10:34:04|00081|bridge|INFO|created port vnet1 on bridge ovsbr0 >> Mar 28 10:34:07|00082|netdev_linux|WARN|ethtool command ETHTOOL_GSET on >> network device vnet1 failed: No such device >> Mar 28 10:34:07|00083|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on vnet1 device >> failed: No such device >> Mar 28 10:34:07|00084|netdev|WARN|failed to get flags for network device >> vnet1: No such device >> Mar 28 10:34:07|00085|netdev|WARN|failed to retrieve MTU for network device >> vnet1: No such device >> Mar 28 10:34:07|00086|netdev|WARN|failed to get flags for network device >> vnet1: No such device >> Mar 28 10:34:07|00087|bridge|INFO|destroyed port vnet1 on bridge ovsbr0 I attempted to run the command but I never set a password for the oneadmin user, but I don't think it's a permissions / sudo access problem. Not really sure where to look next. Any ideas are appreciated. CEPH: I'm trying to use a Ceph datastore with a RBD instead of a cephFS (it's an option). When I try to create a Ceph datastore with a RBD type, I get a state of "Error", but I'm not sure where to look for relevant logs, oned.log didn't seem to have anything, or maybe I'm just grepping for the wrong string. As a work around, I have been creating the directory, creating the rbd then manually mounting it, this seems to work in my test environment, but doesn't seem very scalable, how are others using CEPH? Documentation: I've noticed some errors in the documentation, namely the location of the install scripts, The docs state they are in: >> /usr/share/one/install_gems >> /usr/share/one/sunstone/install_novnc.sh However, I found them in: >> /usr/share/opennebula/install_gems >> /usr/share/opennebula/install_novnc.sh >> /usr/share/opennebula/sunstone/install_novnc.sh Is there some repository of the documentation somewhere that we can contribute to? It's a small thing, but when I'm going through the instructions, I like to copy / paste; I figured it out, but I know if it caused me problems, it might cause others problems too. Thanks again, I can't wait for the final release of OpenNebula! Best Regards, Jon A _______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- Jaime Melis Project Engineer OpenNebula - The Open Source Toolkit for Cloud Computing www.OpenNebula.org | jme...@opennebula.org -- Jaime Melis Project Engineer OpenNebula - The Open Source Toolkit for Cloud Computing www.OpenNebula.org | jme...@opennebula.org _______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message in error, immediately notify the sender and destroy all copies. _______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
