Adding this to ceph.conf on kvm host adds another 2000 iops (20.000
iop/s with one VM). I'm sure most of them are useless on a client kvm /
rbd host but i don't know which one makes sense ;-)
[global]
debug ms = 0/0
debug rbd = 0/0
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
[client]
debug ms = 0/0
debug rbd = 0/0
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
Stefan
Am 12.11.2012 15:35, schrieb Alexandre DERUMIER:
Another idea,
do you have tried to put
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
in a ceph.conf on your kvm host ?
----- Mail original -----
De: "Alexandre DERUMIER" <aderum...@odiso.com>
À: "Stefan Priebe - Profihost AG" <s.pri...@profihost.ag>
Cc: pve-de...@pve.proxmox.com
Envoyé: Lundi 12 Novembre 2012 15:26:36
Objet: Re: [pve-devel] less cores more iops / speed
Maybe some tracing on kvm process could give us clues to find where the cpu is
used ?
Also another idea, can you try with "auth supported=none" ? maybe they are some
overhead with ceph authenfication ?
----- Mail original -----
De: "Alexandre DERUMIER" <aderum...@odiso.com>
À: "Stefan Priebe - Profihost AG" <s.pri...@profihost.ag>
Cc: pve-de...@pve.proxmox.com
Envoyé: Lundi 12 Novembre 2012 15:20:07
Objet: Re: [pve-devel] less cores more iops / speed
Ok thanks.
Seem to use a lot of cpu vs nfs,iscsi ...
I hope that ceph dev will work on this soon !
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.pri...@profihost.ag>
À: "Alexandre DERUMIER" <aderum...@odiso.com>
Cc: "eric" <e...@netwalk.com>, pve-de...@pve.proxmox.com
Envoyé: Lundi 12 Novembre 2012 15:05:08
Objet: Re: [pve-devel] less cores more iops / speed
Am 12.11.2012 13:49, schrieb Alexandre DERUMIER:
One VM on one Host: 18.000 IOP/s
Two VM on one Host: 2x11.000 IOP/s
Three VM on one Host: 3x7.000 IOP/s
And host cpu is 100% ?
No. For three VMs yes. For one and two no. I think librbd / rbd
implementation in kvm is the bottleneck here.
Stefan
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.pri...@profihost.ag>
À: "Alexandre DERUMIER" <aderum...@odiso.com>
Cc: "eric" <e...@netwalk.com>, pve-de...@pve.proxmox.com
Envoyé: Lundi 12 Novembre 2012 12:58:35
Objet: Re: [pve-devel] less cores more iops / speed
Am 12.11.2012 08:51, schrieb Alexandre DERUMIER:
Right now RBD in KVM is limited by CPU speed.
Good to known, so it's seem lack of threading, or maybe somes locks. (so faster
cpu give more iops).
If you lauch parallel fio on same host on different guest, do you get more
total iops ? (for me it's scale)
One VM on one Host: 18.000 IOP/s
Two VM on one Host: 2x11.000 IOP/s
Three VM on one Host: 3x7.000 IOP/s
if you launch 2 parallel fio, on same guest (on differents disk), do you get
more iops ? (for me, it doesn't scale, so raid0 in guest doesn't help).
No it doesn't scale.
Stefan
----- Mail original -----
De: "Stefan Priebe" <s.pri...@profihost.ag>
À: "Alexandre DERUMIER" <aderum...@odiso.com>
Cc: "eric" <e...@netwalk.com>, pve-de...@pve.proxmox.com
Envoyé: Dimanche 11 Novembre 2012 13:07:36
Objet: Re: [pve-devel] less cores more iops / speed
Am 11.11.2012 12:12, schrieb Alexandre DERUMIER:
If I remember good, stefan can achieve 100.000 iops with iscsi with same kvm
host.
Correct but this was always with scsi-generic and I/O multipathing on
host. rbd does not support scsi-generic ;-(
I have checked ceph mailing, stefan seem to have resolved his problem with dual
core with bios update !
Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on
Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz
Single Xeon vs. 2.5Ghz Dual Xeon.
Right now RBD in KVM is limited by CPU speed.
Greets,
Stefan
_______________________________________________
pve-devel mailing list
pve-de...@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-de...@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
