[ceph-users] performance exporting RBD over NFS
Hi, I want to export rbd over nfs in a 10Gb network. Server and Client are DELL R620 with 10Gb nics. rbd cache is disabled ont the server. NFS server write bandwith on his rbd is 1196MB/s NFS client write bandwith on the rbd export is only 233MB/s. NFS client write bandwith on a "local-server-disk" export is 839MB/s my bench is: fio --time_based --name=benchmark --size=20G --runtime=30 --filename=/video1/fiobench --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=write --blocksize=256k --group_reporting my export: /video1 X.X.X.X(rw,sync,no_root_squash) mount : type nfs (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=X.X.X.X,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=X.X.X.X) rbd: rbd image 'video1': size 5120 GB in 1310720 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.1c9dc674b0dc51 format: 2 features: layering flags: My conclusion: - rbd write performance is good - nfs write permormance is good - nfs write on rbd performance is bad do you encounter the same problem ? Marc ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] a big cluster or several small
Hi, Hello, Currently we have a 294 OSD (21 hosts/3 racks) cluster with RBD clients only, 1 single pool (size=3). We want to divide this cluster into several to minimize the risk in case of failure/crash. For example, a cluster for the mail, another for the file servers, a test cluster ... Do you think it's a good idea ? Do you have experience feedback on multiple clusters in production on the same hardware: - containers (LXD or Docker) - multiple cluster on the same host without virtualization (with ceph-deploy ... --cluster ...) - multilple pools ... Do you have any advice? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Infernalis: best practices to start/stop
the documentation is good but it doesn’t work on my centos7: root@cephrr1n8:/root > systemctl status "ceph*" ceph\x2a.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) maybe a bug with centos’s systemd release, is there anybody with centos 7 + infernalis ? > Le 26 nov. 2015 à 13:15, Daniel Swarbrick > a écrit : > > SUSE has pretty good documentation about interacting with Ceph using > systemctl - > https://www.suse.com/documentation/ses-1/book_storage_admin/data/ceph_operating_services.html > > The following should work: > > systemctl start ceph-osd* > > On 26/11/15 12:46, Marc Boisis wrote: >> >> Hi, >> >> I want to know what are the best practices to start or stop all OSDs of a >> node with infernalis. >> Before with init, we used « /etc/init.d/ceph start » now with systemd I >> have a script per osd : "systemctl start ceph-osd@171.service" >> Where is the global one ? >> Thanks in advance! >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Infernalis: best practices to start/stop
Hi, I want to know what are the best practices to start or stop all OSDs of a node with infernalis. Before with init, we used « /etc/init.d/ceph start » now with systemd I have a script per osd : "systemctl start ceph-osd@171.service" Where is the global one ? Thanks in advance! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem with rbd map
Thanks a lot it’s good ROOT:bjorn:/root > rbd create foo --pool pool_ulr_1 --size 512000 -m helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring ROOT:bjorn:/root > rbd map foo --pool pool_ulr_1 --name client.admin -m helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0 ROOT:bjorn:/root > > Le 12 mars 2015 à 13:42, Ilya Dryomov a écrit : > > On Thu, Mar 12, 2015 at 3:33 PM, Marc Boisis wrote: >> I’m trying to create my first ceph disk from a client named bjorn : >> >> [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k >> /etc/ceph/ceph.client.admin.keyring >> [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m >> helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring >> rbd: sysfs write failed >> rbd: map failed: (2) No such file or directory >> >> Can you help me to find the problem ? > > Ah, you are creating an image an a standard pool ("rbd"), but trying to > map from a custom pool ("pool_ulr_1") - hence the -ENOENT. > > Thanks, > >Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem with rbd map
In dmesg: [ 5981.113104] libceph: client14929 fsid cd7dd0a4-075c-4317-8aed-0758085ea9d2 [ 5981.115853] libceph: mon0 10.10.10.64:6789 session established My systems are RHEL 7 with 3.10.0-229.el7.x86_64 kernel > > On Thu, Mar 12, 2015 at 3:33 PM, Marc Boisis wrote: >> I’m trying to create my first ceph disk from a client named bjorn : >> >> [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k >> /etc/ceph/ceph.client.admin.keyring >> [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m >> helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring >> rbd: sysfs write failed >> rbd: map failed: (2) No such file or directory >> >> Can you help me to find the problem ? > > Which kernel is this? Is there anything in dmesg? > > Thanks, > >Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] problem with rbd map
I’m trying to create my first ceph disk from a client named bjorn : [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k /etc/ceph/ceph.client.admin.keyring [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring rbd: sysfs write failed rbd: map failed: (2) No such file or directory Can you help me to find the problem ? [ceph@bjorn ~]$ ceph -v ceph version 0.87.1 (283c2e7cfa2457799f534744d7d549f83ea1335e) [ceph@bjorn ~]$ ceph -s cluster cd7dd0a4-075c-4317-8aed-0758085ea9d2 health HEALTH_OK monmap e5: 5 mons at {borg=10.10.10.58:6789/0,floki=10.10.10.57:6789/0,helga=10.10.10.64:6789/0,horik=10.10.10.60:6789/0,siggy=10.10.10.59:6789/0}, election epoch 88, quorum 0,1,2,3,4 floki,borg,siggy,horik,helga osdmap e732: 60 osds: 60 up, 60 in pgmap v2352: 4160 pgs, 2 pools, 131 bytes data, 2 objects 4145 MB used, 218 TB / 218 TB avail 4160 active+clean [ceph@bjorn ~]$ lsmod | grep rbd rbd73133 0 libceph 235953 1 rbd with strace: open("/sys/bus/rbd/add_single_major", O_WRONLY) = 4 write(4, "10.10.10.64:6789 name=admin,key="..., 61) = -1 ENOENT (No such file or directory) close(4)= 0 write(2, "rbd: sysfs write failed", 23rbd: sysfs write failed) = 23 write(2, "\n", 1 ) = 1 close(3)= 0 write(2, "rbd: map failed: ", 17rbd: map failed: ) = 17 write(2, "(2) No such file or directory", 29(2) No such file or directory) = 29 write(2, "\n", 1 ) = 1 exit_group(2) = ? +++ exited with 2 +++ [ceph@bjorn ~]$ ll /sys/bus/rbd/add_single_major --w--- 1 root root 4096 Mar 12 12:01 /sys/bus/rbd/add_single_major thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com