optmize librbd for iops
Hello list, are there any plans to optimize librbd for iops? Right now i'm able to get 50.000 iop/s via iscsi and 100.000 iop/s using multipathing with iscsi. With librbd i'm stuck to around 18.000iops. As this scales with more hosts but not with more disks in a vm. It must be limited by rbd implementation in kvm / librbd. Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Disabling journal
On Sun, 11 Nov 2012, Stefan Priebe wrote: > Hi Sage, > > > With btrfs, yes, although this isn't something we have tested in a while. > I'm not using btrfs as long as the devs claim it is not ready for prod. In that case, the journal is needed for consistency of the fs; we rely on writeahead journaling. It can't be turned off. Putting it on a ramdisk in this case is interesting for performance, but it means that a crash/reboot/powerloss event leaves the fs in an inconsistent and unusable state. The only time tmpfs is potentially useful in production is when you're using btrfs *and* have independent backup power sources for replicas (and can thus avoid worrying about a site-wide power failure and loss of journal). (Or have relaxed requirements for the durability of recent writes.) sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Disabling journal
Am 12.11.2012 15:42, schrieb Sage Weil: On Sun, 11 Nov 2012, Stefan Priebe wrote: Hi Sage, With btrfs, yes, although this isn't something we have tested in a while. I'm not using btrfs as long as the devs claim it is not ready for prod. In that case, the journal is needed for consistency of the fs; we rely on writeahead journaling. It can't be turned off. Putting it on a ramdisk in this case is interesting for performance, but it means that a crash/reboot/powerloss event leaves the fs in an inconsistent and unusable state. But only if for replicas 2 both nodes crash / have a powerloss? The only time tmpfs is potentially useful in production is when you're using btrfs *and* have independent backup power sources for replicas (and can thus avoid worrying about a site-wide power failure and loss of journal). (Or have relaxed requirements for the durability of recent writes.) What happens for XFS and replicas two and ONE host has a power loss? The other replica / journal should be still there. I've no idea where to put the journal on. I mean i've 8 SSDs per Host one per osd each with a write IOP/s speed of 45.000 iops to whole IOP/s write speed of 360.000 IOP/s per Node. Which journal device can handle this? And if i put the journal on the same disk as the OSD it has to copy the data around. Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Disabling journal
On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote: > Am 12.11.2012 15:42, schrieb Sage Weil: > > On Sun, 11 Nov 2012, Stefan Priebe wrote: > > > Hi Sage, > > > > > > > With btrfs, yes, although this isn't something we have tested in a > > > > while. > > > I'm not using btrfs as long as the devs claim it is not ready for prod. > > > > In that case, the journal is needed for consistency of the fs; we rely on > > writeahead journaling. It can't be turned off. > > > > Putting it on a ramdisk in this case is interesting for performance, but > > it means that a crash/reboot/powerloss event leaves the fs in an > > inconsistent and unusable state. > > But only if for replicas 2 both nodes crash / have a powerloss? Then you're okay.. but the one that lost the journal effectively also lost the contents of the SSD. Also, manual intervention is currently needed to reinitialize the osd (since this is not a normal failure mode). > > The only time tmpfs is potentially useful in production is when you're > > using btrfs *and* have independent backup power sources for replicas (and > > can thus avoid worrying about a site-wide power failure and loss of > > journal). (Or have relaxed requirements for the durability of recent > > writes.) > What happens for XFS and replicas two and ONE host has a power loss? The other > replica / journal should be still there. > > I've no idea where to put the journal on. > > I mean i've 8 SSDs per Host one per osd each with a write IOP/s speed of > 45.000 iops to whole IOP/s write speed of 360.000 IOP/s per Node. > > Which journal device can handle this? And if i put the journal on the same > disk as the OSD it has to copy the data around. I think you have two choices. Either put the journal SSDs (perhaps a journal on an existing one), or use a higher-end NVRAM-based device. There are several of these out there, although I'm blanking on product names at the moment. The best are probably the battery-backed DRAM ones with a bit of flash for when the battery gets low. Lots of RAID controllers also have some onboard NVRAM that can often be finagled into being useful, at least with spinning disks; I'm not sure how they perform with SSDs. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] ceph-mon crashes
Hi Stefan, Any chance you can get me a larger chunk of the log from the monitor that was the leader by the time you issued those commands until the point the monitor crashed (from the excerpt you provided, that should be mon.b)? -Joao On 11/12/2012 06:21 AM, Stefan Priebe - Profihost AG wrote: > > Am 12.11.2012 um 00:26 schrieb Joao Eduardo Luis : > >> On 11/11/2012 10:22 PM, Stefan Priebe wrote: >>> I wanted to remove osd.51 to 54. And executed the following commands: >>> ceph osd 51 out >>> ceph osd 52 out >>> ceph osd 53 out >>> ceph osd 54 out >>> >>> Greets, >>> Stefan >> >> Were those osds shutdown/killed before issuing the out commands? > No > > >> >> And I'm assuming all the commands completed successfully with a 'marked >> out osd.X'. Is this correct? > Yes > > Stefan > > >> >> -Joao >> >> >>> Am 11.11.2012 23:20, schrieb Joao Eduardo Luis: Hi Stefan, Any chance you can get us the output of `ceph osd dump | grep 'osd.53'`? Also, do you have any idea what happened to osd.53? -Joao On 11/11/2012 09:51 PM, Stefan Priebe wrote: > Hello list, > > i've now seen the following ceph-mon crash several times: >-22> 2012-11-11 22:45:10.941641 7f5d49168700 1 > mon.b@1(leader).paxos(pgmap active c 11425..11925) is_readable > now=2012-11-11 22:45:10.941643 lease_expire=2012-11-11 22:45:15.313577 > has v1017 lc 11925 >-21> 2012-11-11 22:45:11.017199 7f5d49168700 1 -- > 10.255.0.101:6789/0 <== mon.2 10.255.0.102:6789/0 123 paxos(logm > lease_ack lc 14415 fc 13913 pn 0 opn 0 gv {}) v2 88+0+0 (1926207536 > 0 0) 0x58fe840 con 0x2c38c60 >-20> 2012-11-11 22:45:11.084775 7f5d49168700 1 -- > 10.255.0.101:6789/0 <== mon.2 10.255.0.102:6789/0 124 paxos(mdsmap > lease_ack lc 1 fc 1 pn 0 opn 0 gv {}) v2 88+0+0 (2037515134 0 0) > 0x4e59080 con 0x2c38c60 >-19> 2012-11-11 22:45:11.084812 7f5d49168700 1 -- > 10.255.0.101:6789/0 <== mon.2 10.255.0.102:6789/0 125 paxos(monmap > lease_ack lc 1 fc 1 pn 0 opn 0 gv {}) v2 88+0+0 (3822314369 0 0) > 0x4e58580 con 0x2c38c60 >-18> 2012-11-11 22:45:11.230970 7f5d48065700 2 -- > 10.255.0.101:6789/0 >> 10.255.0.100:6789/0 pipe(0x2cc9b40 sd=17 :6789 > pgs=16 cs=2 l=0).connect error 10.255.0.100:6789/0, 111: Connection > refused >-17> 2012-11-11 22:45:11.231004 7f5d48065700 2 -- > 10.255.0.101:6789/0 >> 10.255.0.100:6789/0 pipe(0x2cc9b40 sd=17 :6789 > pgs=16 cs=2 l=0).fault 111: Connection refused >-16> 2012-11-11 22:45:11.238427 7f5d49168700 1 -- > 10.255.0.101:6789/0 <== osd.33 10.255.0.102:6806/3537 1315 > pg_stats(478 pgs tid 185 v 1017) v1 158856+0+0 (1263639313 0 0) > 0x2ca5b40 con 0x2ca4dc0 >-15> 2012-11-11 22:45:11.238451 7f5d49168700 1 > mon.b@1(leader).paxos(pgmap active c 11425..11925) is_readable > now=2012-11-11 22:45:11.238452 lease_expire=2012-11-11 22:45:15.313577 > has v1017 lc 11925 >-14> 2012-11-11 22:45:11.287036 7f5d49969700 5 > mon.b@1(leader).paxos(pgmap active c 11425..11925) propose_new_value > 11926 311714 bytes, gv 28550 >-13> 2012-11-11 22:45:11.296084 7f5d49168700 1 -- > 10.255.0.101:6789/0 <== osd.41 10.255.0.103:6800/3324 1086 > pg_stats(440 pgs tid 185 v 1017) v1 146276+0+0 (1903337195 0 0) > 0x39bad80 con 0x2ca4c60 >-12> 2012-11-11 22:45:11.350697 7f5d49969700 1 -- > 10.255.0.101:6789/0 --> mon.2 10.255.0.102:6789/0 -- paxos(pgmap begin > lc 11925 fc 0 pn 3501 opn 0 gv {11926=28550}) v2 -- ?+0 0x4e60b00 >-11> 2012-11-11 22:45:11.350790 7f5d49168700 1 > mon.b@1(leader).paxos(pgmap updating c 11425..11925) is_readable > now=2012-11-11 22:45:11.350793 lease_expire=2012-11-11 22:45:15.313577 > has v1017 lc 11925 >-10> 2012-11-11 22:45:11.435570 7f5d49168700 1 -- > 10.255.0.101:6789/0 <== mon.2 10.255.0.102:6789/0 126 paxos(pgmap > accept lc 11925 fc 0 pn 3501 opn 0 gv {}) v2 88+0+0 (3719346061 0 > 0) 0x4e60b00 con 0x2c38c60 > -9> 2012-11-11 22:45:11.476091 7f5d49168700 1 -- > 10.255.0.101:6789/0 --> mon.2 10.255.0.102:6789/0 -- paxos(pgmap commit > lc 11926 fc 0 pn 3501 opn 0 gv {11926=28550}) v2 -- ?+0 0x4e58580 > -8> 2012-11-11 22:45:11.476125 7f5d49168700 1 -- > 10.255.0.101:6789/0 --> mon.2 10.255.0.102:6789/0 -- paxos(pgmap lease > lc 11926 fc 11425 pn 0 opn 0 gv {}) v2 -- ?+0 0x4e59080 > -7> 2012-11-11 22:45:11.593945 7f5d49168700 0 log [INF] : pgmap > v11926: 7032 pgs: 5977 active+clean, 62 active+remapped+wait_backfill, > 586 active+degraded+wait_backfill, 32 active+remapped+backfilling, 18 > active+degraded+backfilling, 343 active+degraded+remapped+wait_backfill, > 2 remapped+peering, 12 active+degraded+remapped+backfilling; 61211 MB > data, 61662 MB used, 4185 GB / 4245 GB avail; 5419/31221 degraded > (
ceph cluster hangs when rebooting one node
Hello list, i was checking what happens if i reboot a ceph node. Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is possible. ceph -w: Looks like this: 2012-11-12 16:03:58.191106 mon.0 [INF] pgmap v19013: 7032 pgs: 7032 active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:08.365557 mon.0 [INF] mon.a calling new monitor election 2012-11-12 16:04:13.422682 mon.0 [INF] mon.a@0 won leader election with quorum 0,2 2012-11-12 16:04:13.708045 mon.0 [INF] pgmap v19014: 7032 pgs: 7032 active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:13.708059 mon.0 [INF] mdsmap e1: 0/0/1 up 2012-11-12 16:04:13.708070 mon.0 [INF] osdmap e4582: 20 osds: 20 up, 20 in 2012-11-12 16:04:08.242688 mon.2 [INF] mon.c calling new monitor election 2012-11-12 16:04:13.708089 mon.0 [INF] monmap e1: 3 mons at {a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0} 2012-11-12 16:04:14.070593 mon.0 [INF] pgmap v19015: 7032 pgs: 7032 active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:15.283954 mon.0 [INF] pgmap v19016: 7032 pgs: 7032 active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:18.506812 mon.0 [INF] osd.21 10.255.0.101:6800/5049 failed (3 reports from 3 peers after 20.339769 >= grace 20.00) 2012-11-12 16:04:18.890003 mon.0 [INF] osdmap e4583: 20 osds: 19 up, 20 in 2012-11-12 16:04:19.137936 mon.0 [INF] pgmap v19017: 7032 pgs: 6720 active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:20.024595 mon.0 [INF] osdmap e4584: 20 osds: 19 up, 20 in 2012-11-12 16:04:20.330149 mon.0 [INF] pgmap v19018: 7032 pgs: 6720 active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:21.535471 mon.0 [INF] pgmap v19019: 7032 pgs: 6720 active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail 2012-11-12 16:04:24.181292 mon.0 [INF] osd.22 10.255.0.101:6803/5153 failed (3 reports from 3 peers after 23.013550 >= grace 20.00) 2012-11-12 16:04:24.182208 mon.0 [INF] osd.23 10.255.0.101:6806/5276 failed (3 reports from 3 peers after 21.000834 >= grace 20.00) 2012-11-12 16:04:24.671373 mon.0 [INF] pgmap v19020: 7032 pgs: 6637 active+clean, 208 stale+active+clean, 187 incomplete; 91615 MB data, 174 GB used, 4295 GB / 4469 GB avail 2012-11-12 16:04:24.829022 mon.0 [INF] osdmap e4585: 20 osds: 17 up, 20 in 2012-11-12 16:04:24.870969 mon.0 [INF] osd.24 10.255.0.101:6809/5397 failed (3 reports from 3 peers after 20.688672 >= grace 20.00) 2012-11-12 16:04:25.522333 mon.0 [INF] pgmap v19021: 7032 pgs: 5912 active+clean, 933 stale+active+clean, 187 incomplete; 91615 MB data, 174 GB used, 4295 GB / 4469 GB avail 2012-11-12 16:04:25.596927 mon.0 [INF] osd.24 10.255.0.101:6809/5397 failed (3 reports from 3 peers after 21.708444 >= grace 20.00) 2012-11-12 16:04:26.077545 mon.0 [INF] osdmap e4586: 20 osds: 16 up, 20 in 2012-11-12 16:04:26.606475 mon.0 [INF] pgmap v19022: 7032 pgs: 5394 active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 173 GB used, 4296 GB / 4469 GB avail 2012-11-12 16:04:27.162034 mon.0 [INF] osdmap e4587: 20 osds: 16 up, 20 in 2012-11-12 16:04:27.656974 mon.0 [INF] pgmap v19023: 7032 pgs: 5394 active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 173 GB used, 4296 GB / 4469 GB avail 2012-11-12 16:04:30.229958 mon.0 [INF] pgmap v19024: 7032 pgs: 5394 active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 172 GB used, 4296 GB / 4469 GB avail 2012-11-12 16:04:31.411989 mon.0 [INF] pgmap v19025: 7032 pgs: 5394 active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 172 GB used, 4296 GB / 4469 GB avail 2012-11-12 16:04:32.617576 mon.0 [INF] pgmap v19026: 7032 pgs: 4660 active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB / 4469 GB avail 2012-11-12 16:04:35.172861 mon.0 [INF] pgmap v19027: 7032 pgs: 4660 active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB / 4469 GB avail 2012-11-12 16:04:30.505872 osd.53 [WRN] 6 slow requests, 6 included below; oldest blocked for > 30.247691 secs 2012-11-12 16:04:30.505875 osd.53 [WRN] slow request 30.247691 seconds old, received at 2012-11-12 16:04:00.258118: osd_op(client.131626.0:771962 rb.0.107a.734602d5.0bce [write 2478080~4096] 3.562a9efc) v4 currently reached pg 2012-11-12 16:04:30.505879 osd.53 [WRN] slow request 30.238016 seconds old, received at 2012-11-12 16:04:00.267793: osd_op(client.131626.0:772116 rb.0.107a.734602d5.1608 [write 262144~4096] 3.a47890e) v4 currently reached pg 2012-11-12 16:04:30.505881 osd.53 [WRN] slow request 30.236572 seconds old, received at 2012-11-12 16:04:00.269237: osd_op(client.131626.0:772141 rb.0.107a.734602d5.1777 [write 798720~4096] 3.547bc855) v4 currently reached pg 2012-11-12 16:04:30.505883 osd.53 [WRN] slow reque
Re: [BUG] ceph-mon crashes
Am 12.11.2012 15:58, schrieb Joao Eduardo Luis: Hi Stefan, Any chance you can get me a larger chunk of the log from the monitor that was the leader by the time you issued those commands until the point the monitor crashed (from the excerpt you provided, that should be mon.b)? Sure: https://www.dropbox.com/s/8e604bihk56m0yd/ceph-mon.b.log.1.gz Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ceph cluster hangs when rebooting one node
On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote: > Hello list, > > i was checking what happens if i reboot a ceph node. > > Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is > possible. If you are using the current master, the new 'min_size' may be biting you; ceph osd dump | grep ^pool and see if you see min_size for your pools. You can change that back to the norma behavior with ceph osd pool set min_size 1 sage > > ceph -w: > Looks like this: > 2012-11-12 16:03:58.191106 mon.0 [INF] pgmap v19013: 7032 pgs: 7032 > active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail > 2012-11-12 16:04:08.365557 mon.0 [INF] mon.a calling new monitor election > 2012-11-12 16:04:13.422682 mon.0 [INF] mon.a@0 won leader election with quorum > 0,2 > 2012-11-12 16:04:13.708045 mon.0 [INF] pgmap v19014: 7032 pgs: 7032 > active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail > 2012-11-12 16:04:13.708059 mon.0 [INF] mdsmap e1: 0/0/1 up > 2012-11-12 16:04:13.708070 mon.0 [INF] osdmap e4582: 20 osds: 20 up, 20 in > 2012-11-12 16:04:08.242688 mon.2 [INF] mon.c calling new monitor election > 2012-11-12 16:04:13.708089 mon.0 [INF] monmap e1: 3 mons at > {a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0} > 2012-11-12 16:04:14.070593 mon.0 [INF] pgmap v19015: 7032 pgs: 7032 > active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail > 2012-11-12 16:04:15.283954 mon.0 [INF] pgmap v19016: 7032 pgs: 7032 > active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail > 2012-11-12 16:04:18.506812 mon.0 [INF] osd.21 10.255.0.101:6800/5049 failed (3 > reports from 3 peers after 20.339769 >= grace 20.00) > 2012-11-12 16:04:18.890003 mon.0 [INF] osdmap e4583: 20 osds: 19 up, 20 in > 2012-11-12 16:04:19.137936 mon.0 [INF] pgmap v19017: 7032 pgs: 6720 > active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB / > 4469 GB avail > 2012-11-12 16:04:20.024595 mon.0 [INF] osdmap e4584: 20 osds: 19 up, 20 in > 2012-11-12 16:04:20.330149 mon.0 [INF] pgmap v19018: 7032 pgs: 6720 > active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB / > 4469 GB avail > 2012-11-12 16:04:21.535471 mon.0 [INF] pgmap v19019: 7032 pgs: 6720 > active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB / > 4469 GB avail > 2012-11-12 16:04:24.181292 mon.0 [INF] osd.22 10.255.0.101:6803/5153 failed (3 > reports from 3 peers after 23.013550 >= grace 20.00) > 2012-11-12 16:04:24.182208 mon.0 [INF] osd.23 10.255.0.101:6806/5276 failed (3 > reports from 3 peers after 21.000834 >= grace 20.00) > 2012-11-12 16:04:24.671373 mon.0 [INF] pgmap v19020: 7032 pgs: 6637 > active+clean, 208 stale+active+clean, 187 incomplete; 91615 MB data, 174 GB > used, 4295 GB / 4469 GB avail > 2012-11-12 16:04:24.829022 mon.0 [INF] osdmap e4585: 20 osds: 17 up, 20 in > 2012-11-12 16:04:24.870969 mon.0 [INF] osd.24 10.255.0.101:6809/5397 failed (3 > reports from 3 peers after 20.688672 >= grace 20.00) > 2012-11-12 16:04:25.522333 mon.0 [INF] pgmap v19021: 7032 pgs: 5912 > active+clean, 933 stale+active+clean, 187 incomplete; 91615 MB data, 174 GB > used, 4295 GB / 4469 GB avail > 2012-11-12 16:04:25.596927 mon.0 [INF] osd.24 10.255.0.101:6809/5397 failed (3 > reports from 3 peers after 21.708444 >= grace 20.00) > 2012-11-12 16:04:26.077545 mon.0 [INF] osdmap e4586: 20 osds: 16 up, 20 in > 2012-11-12 16:04:26.606475 mon.0 [INF] pgmap v19022: 7032 pgs: 5394 > active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 173 GB > used, 4296 GB / 4469 GB avail > 2012-11-12 16:04:27.162034 mon.0 [INF] osdmap e4587: 20 osds: 16 up, 20 in > 2012-11-12 16:04:27.656974 mon.0 [INF] pgmap v19023: 7032 pgs: 5394 > active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 173 GB > used, 4296 GB / 4469 GB avail > 2012-11-12 16:04:30.229958 mon.0 [INF] pgmap v19024: 7032 pgs: 5394 > active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 172 GB > used, 4296 GB / 4469 GB avail > 2012-11-12 16:04:31.411989 mon.0 [INF] pgmap v19025: 7032 pgs: 5394 > active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 172 GB > used, 4296 GB / 4469 GB avail > 2012-11-12 16:04:32.617576 mon.0 [INF] pgmap v19026: 7032 pgs: 4660 > active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB / 4469 GB > avail > 2012-11-12 16:04:35.172861 mon.0 [INF] pgmap v19027: 7032 pgs: 4660 > active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB / 4469 GB > avail > 2012-11-12 16:04:30.505872 osd.53 [WRN] 6 slow requests, 6 included below; > oldest blocked for > 30.247691 secs > 2012-11-12 16:04:30.505875 osd.53 [WRN] slow request 30.247691 seconds old, > received at 2012-11-12 16:04:00.258118: osd_op(client.131626.0:771962 > rb.0.107a.734602d5.0bce [write 2478080~4096] 3.562a9efc) v4 currently > reached pg > 2012-11-12 16:04:30.505879 osd.53 [WRN] slow request 30.238016 seconds old, > received at 2012-11-12 16:04:00.2
Re: ceph cluster hangs when rebooting one node
Am 12.11.2012 16:11, schrieb Sage Weil: On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i was checking what happens if i reboot a ceph node. Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is possible. If you are using the current master, the new 'min_size' may be biting you; ceph osd dump | grep ^pool and see if you see min_size for your pools. You can change that back to the norma behavior with No i don't see any min size: # ceph osd dump | grep ^pool pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 1344 pgp_num 1344 last_change 1 owner 0 crash_replay_interval 45 pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 1344 pgp_num 1344 last_change 1 owner 0 pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 1344 pgp_num 1344 last_change 1 owner 0 pool 3 'kvmpool1' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 3000 pgp_num 3000 last_change 958 owner 0 ceph osd pool set min_size 1 Yes this helps! But min_size is still not shown in ceph osd dump. Also when i reboot a node it takes up to 10s-20s until all osds from this node are set to failed and the I/O starts again. Should i issue an ceph osd out command before? But i had already this set for all my rules in my crushmap min_size 1 max_size 2 in my crushmap for each rule. Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [pve-devel] less cores more iops / speed
Adding this to ceph.conf on kvm host adds another 2000 iops (20.000 iop/s with one VM). I'm sure most of them are useless on a client kvm / rbd host but i don't know which one makes sense ;-) [global] debug ms = 0/0 debug rbd = 0/0 debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 [client] debug ms = 0/0 debug rbd = 0/0 debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 Stefan Am 12.11.2012 15:35, schrieb Alexandre DERUMIER: Another idea, do you have tried to put debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 in a ceph.conf on your kvm host ? - Mail original - De: "Alexandre DERUMIER" À: "Stefan Priebe - Profihost AG" Cc: pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 15:26:36 Objet: Re: [pve-devel] less cores more iops / speed Maybe some tracing on kvm process could give us clues to find where the cpu is used ? Also another idea, can you try with "auth supported=none" ? maybe they are some overhead with ceph authenfication ? - Mail original - De: "Alexandre DERUMIER" À: "Stefan Priebe - Profihost AG" Cc: pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 15:20:07 Objet: Re: [pve-devel] less cores more iops / speed Ok thanks. Seem to use a lot of cpu vs nfs,iscsi ... I hope that ceph dev will work on this soon ! - Mail original - De: "Stefan Priebe - Profihost AG" À: "Alexandre DERUMIER" Cc: "eric" , pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 15:05:08 Objet: Re: [pve-devel] less cores more iops / speed Am 12.11.2012 13:49, schrieb Alexandre DERUMIER: One VM on one Host: 18.000 IOP/s Two VM on one Host: 2x11.000 IOP/s Three VM on one Host: 3x7.000 IOP/s And host cpu is 100% ? No. For three VMs yes. For one and two no. I think librbd / rbd implementation in kvm is the bottleneck here. Stefan - Mail original - De: "Stefan Priebe - Profihost AG" À: "Alexandre DERUMIER" Cc: "eric" , pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 12:58:35 Objet: Re: [pve-devel] less cores more iops / speed Am 12.11.2012 08:51, schrieb Alexandre DERUMIER: Right now RBD in KVM is limited by CPU speed. Good to known, so it's seem lack of threading, or maybe somes locks. (so faster cpu give more iops). If you lauch parallel fio on same host on different guest, do you get more total iops ? (for me it's scale) One VM on one Host: 18.000 IOP/s Two VM on one Host: 2x11.000 IOP/s Three VM on one Host: 3x7.000 IOP/s if you launch 2 parallel fio, on same guest (on differents disk), do you get more iops ? (for me, it doesn't scale, so raid0 in guest doesn't help). No it doesn't scale. Stefan - Mail original - De: "Stefan Priebe" À: "Alexandre DERUMIER" Cc: "eric" , pve-de...@pve.proxmox.com Envoyé: Dimanche 11 Novembre 2012 13:07:36 Objet: Re: [pve-devel] less cores more iops / speed Am 11.11.2012 12:12, schrieb Alexandre DERUMIER: If I remember good, stefan can achieve 100.000 iops with iscsi with same kvm host. Correct but this was always with scsi-generic and I/O multipathing on host. rbd does not support scsi-generic ;-( I have checked ceph mailing, stefan seem to have resolved his problem with dual core with bios update ! Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz Single Xeon vs. 2.5Ghz Dual Xeon. Right now RBD in KVM is limited by CPU speed. Greets, Stefan ___ pve-devel mailing list pve-de...@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listin
improve speed with auth supported=none
Hello list, i'm still trying to improve ceph speed. Disable logging on host and rbd client gives me additional 5000 iop/s which is great. But i also wanted to try disabling authentication using: auth supported=none How does this work? Do i just have to place this line under global section in ceph.conf? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
changed rbd cp behavior in 0.53
Hi, For this version, rbd cp assumes that destination pool is the same as source, not 'rbd', if pool in the destination path is omitted. rbd cp install/img testimg rbd ls install img testimg Is this change permanent? Thanks! -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] ceph-mon crashes
On 11/12/2012 03:10 PM, Stefan Priebe - Profihost AG wrote: > Am 12.11.2012 15:58, schrieb Joao Eduardo Luis: >> Hi Stefan, >> >> >> Any chance you can get me a larger chunk of the log from the monitor >> that was the leader by the time you issued those commands until the >> point the monitor crashed (from the excerpt you provided, that should be >> mon.b)? > > Sure: > https://www.dropbox.com/s/8e604bihk56m0yd/ceph-mon.b.log.1.gz > > Greets, > Stefan Hi Stefan, Thanks for the log. Can you please confirm me that sometime between you issuing the out command and mon.b failing, you had yet another monitor (maybe mon.a) that was the leader but for some reason it was down by the time that mon.b failed? If so, could you provide the log for that monitor as well, given this log doesn't have some infos I'm looking for? -Joao -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
pull request: ceph-qa-suite branch wip-java
This patch adds a yaml file to add the libcephfs-java tests to the nightly qa test set. Best, -Joe Buck -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: improve speed with auth supported=none
I guess you can refer to that link on the list: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/9776 btw do you get 5000 iop/s on the rbd kernel or on a vm disk? cheers. On Mon, Nov 12, 2012 at 4:37 PM, Stefan Priebe - Profihost AG wrote: > Hello list, > > i'm still trying to improve ceph speed. Disable logging on host and rbd > client gives me additional 5000 iop/s which is great. > > But i also wanted to try disabling authentication using: > auth supported=none > > How does this work? Do i just have to place this line under global section > in ceph.conf? > > Greets, > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] ceph-mon crashes
Hi Joao, Am 12.11.2012 18:05, schrieb Joao Eduardo Luis: Can you please confirm me that sometime between you issuing the out command and mon.b failing, you had yet another monitor (maybe mon.a) that was the leader but for some reason it was down by the time that mon.b failed? If so, could you provide the log for that monitor as well, given this log doesn't have some infos I'm looking for? Not sure but here are the logs of the other two mons: https://www.dropbox.com/s/jztsedvj1b2kjje/ceph-mon.a.log.1.gz https://www.dropbox.com/s/62jkfbbbgvs5o25/ceph-mon.c.log.1.gz Thanks, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [pve-devel] less cores more iops / speed
On 11/12/2012 07:33 AM, Stefan Priebe - Profihost AG wrote: Adding this to ceph.conf on kvm host adds another 2000 iops (20.000 iop/s with one VM). I'm sure most of them are useless on a client kvm / rbd host but i don't know which one makes sense ;-) [global] debug ms = 0/0 debug rbd = 0/0 debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 [client] For the client side you'd these settings to disable all debug logging: [client] debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug filer = 0/0 debug objecter = 0/0 debug rados = 0/0 debug rbd = 0/0 debug objectcacher = 0/0 debug client = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 Josh debug ms = 0/0 debug rbd = 0/0 debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 Stefan Am 12.11.2012 15:35, schrieb Alexandre DERUMIER: Another idea, do you have tried to put debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 in a ceph.conf on your kvm host ? - Mail original - De: "Alexandre DERUMIER" À: "Stefan Priebe - Profihost AG" Cc: pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 15:26:36 Objet: Re: [pve-devel] less cores more iops / speed Maybe some tracing on kvm process could give us clues to find where the cpu is used ? Also another idea, can you try with "auth supported=none" ? maybe they are some overhead with ceph authenfication ? - Mail original - De: "Alexandre DERUMIER" À: "Stefan Priebe - Profihost AG" Cc: pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 15:20:07 Objet: Re: [pve-devel] less cores more iops / speed Ok thanks. Seem to use a lot of cpu vs nfs,iscsi ... I hope that ceph dev will work on this soon ! - Mail original - De: "Stefan Priebe - Profihost AG" À: "Alexandre DERUMIER" Cc: "eric" , pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 15:05:08 Objet: Re: [pve-devel] less cores more iops / speed Am 12.11.2012 13:49, schrieb Alexandre DERUMIER: One VM on one Host: 18.000 IOP/s Two VM on one Host: 2x11.000 IOP/s Three VM on one Host: 3x7.000 IOP/s And host cpu is 100% ? No. For three VMs yes. For one and two no. I think librbd / rbd implementation in kvm is the bottleneck here. Stefan - Mail original - De: "Stefan Priebe - Profihost AG" À: "Alexandre DERUMIER" Cc: "eric" , pve-de...@pve.proxmox.com Envoyé: Lundi 12 Novembre 2012 12:58:35 Objet: Re: [pve-devel] less cores more iops / speed Am 12.11.2012 08:51, schrieb Alexandre DERUMIER: Right now RBD in KVM is limited by CPU speed. Good to known, so it's seem lack of threading, or maybe somes locks. (so faster cpu give more iops). If you lauch parallel fio on same host on different guest, do you get more total iops ? (for me it's scale) One VM on one Host: 18.000 IOP/s Two VM on one Host: 2x11.000 IOP/s Three VM on one Host: 3x7.000 IOP/s if you launch 2 parallel fio, on same guest (on differents disk), do you get more iops ? (for me, it doesn't scale, so raid0 in guest doesn't help). No it doesn't scale. Stefan - Mail original - De: "Stefan Priebe" À: "Alexandre DERUMIER" Cc: "eric" , pve-de...@pve.proxmox.com Envoyé: Dimanche 11 Novembre 2012 13:07:36 Objet: Re: [pve-devel] less cores more iops / speed Am 11.11.2012 12:12, schrieb Alexandre DERUMIER: If I remember good, stefan can achieve 100.000 iops with iscsi with same kvm host. Correct but this was always with scsi-generic and I/O mu
Re: [pve-devel] less cores more iops / speed
Hi Josh, For the client side you'd these settings to disable all debug logging: ... Thanks! Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] ceph-mon crashes
On 11/12/2012 06:30 PM, Stefan Priebe wrote: > Hi Joao, > > Am 12.11.2012 18:05, schrieb Joao Eduardo Luis: >> Can you please confirm me that sometime between you issuing the out >> command and mon.b failing, you had yet another monitor (maybe mon.a) >> that was the leader but for some reason it was down by the time that >> mon.b failed? >> >> If so, could you provide the log for that monitor as well, given this >> log doesn't have some infos I'm looking for? > Not sure but here are the logs of the other two mons: > https://www.dropbox.com/s/jztsedvj1b2kjje/ceph-mon.a.log.1.gz > > https://www.dropbox.com/s/62jkfbbbgvs5o25/ceph-mon.c.log.1.gz > > Thanks, > Stefan Thanks Stefan, I'll be looking into this. For future reference, I created issue #3477 on the tracker: http://tracker.newdream.net/issues/3477 -Joao -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Build regressions/improvements in v3.7-rc5
On Mon, Nov 12, 2012 at 9:58 PM, Geert Uytterhoeven wrote: > JFYI, when comparing v3.7-rc5 to v3.7-rc4[3], the summaries are: > - build errors: +14/-4 14 regressions: + drivers/virt/fsl_hypervisor.c: error: 'MSR_GS' undeclared (first use in this function): => 799:93 + error: No rule to make target drivers/scsi/aic7xxx/aicasm/*.[chyl]: => N/A + net/ceph/ceph_common.c: error: dereferencing pointer to incomplete type: => 272:13 + net/ceph/ceph_common.c: error: implicit declaration of function 'request_key' [-Werror=implicit-function-declaration]: => 249:2 + net/ceph/crypto.c: error: dereferencing pointer to incomplete type: => 463:19, 434:46, 452:5, 448:52, 429:23, 467:36, 447:18 + net/ceph/crypto.c: error: implicit declaration of function 'key_payload_reserve' [-Werror=implicit-function-declaration]: => 437:2 + net/ceph/crypto.c: error: implicit declaration of function 'register_key_type' [-Werror=implicit-function-declaration]: => 481:2 + net/ceph/crypto.c: error: implicit declaration of function 'unregister_key_type' [-Werror=implicit-function-declaration]: => 485:2 + net/ceph/crypto.c: error: unknown field 'destroy' specified in initializer: => 477:2 + net/ceph/crypto.c: error: unknown field 'instantiate' specified in initializer: => 475:2 + net/ceph/crypto.c: error: unknown field 'match' specified in initializer: => 476:2 + net/ceph/crypto.c: error: unknown field 'name' specified in initializer: => 474:2 + net/ceph/crypto.c: error: variable 'key_type_ceph' has initializer but incomplete type: => 473:8 powerpc-randconfig + error: relocation truncated to fit: R_PPC64_REL24 against symbol `._mcount' defined in .text section in arch/powerpc/kernel/entry_64.o: (.text+0x1ff9eb8) => (.text+0x1ffa274), (.text+0x1ff7840) powerpc-allyesconfig > [1] http://kisskb.ellerman.id.au/kisskb/head/5614/ (all 117 configs) > [3] http://kisskb.ellerman.id.au/kisskb/head/5600/ (all 117 configs) Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] ceph-mon crashes
Thanks i'm subscribed to the tracker now. Stefan Am 12.11.2012 21:40, schrieb Joao Eduardo Luis: On 11/12/2012 06:30 PM, Stefan Priebe wrote: Hi Joao, Am 12.11.2012 18:05, schrieb Joao Eduardo Luis: Can you please confirm me that sometime between you issuing the out command and mon.b failing, you had yet another monitor (maybe mon.a) that was the leader but for some reason it was down by the time that mon.b failed? If so, could you provide the log for that monitor as well, given this log doesn't have some infos I'm looking for? Not sure but here are the logs of the other two mons: https://www.dropbox.com/s/jztsedvj1b2kjje/ceph-mon.a.log.1.gz https://www.dropbox.com/s/62jkfbbbgvs5o25/ceph-mon.c.log.1.gz Thanks, Stefan Thanks Stefan, I'll be looking into this. For future reference, I created issue #3477 on the tracker: http://tracker.newdream.net/issues/3477 -Joao -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: improve speed with auth supported=none
Thanks, this gives another burst for iops. I'm now at 23.000 iops ;-) So for random 4k iops ceph auth and especially the logging is a lot of overhead. Greets, Stefan Am 12.11.2012 19:26, schrieb Sébastien Han: I guess you can refer to that link on the list: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/9776 btw do you get 5000 iop/s on the rbd kernel or on a vm disk? cheers. On Mon, Nov 12, 2012 at 4:37 PM, Stefan Priebe - Profihost AG wrote: Hello list, i'm still trying to improve ceph speed. Disable logging on host and rbd client gives me additional 5000 iop/s which is great. But i also wanted to try disabling authentication using: auth supported=none How does this work? Do i just have to place this line under global section in ceph.conf? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it seems we no longer have this hang. On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin wrote: > On 11/08/2012 02:10 PM, Mandell Degerness wrote: >> >> We are seeing a somewhat random, but frequent hang on our systems >> during startup. The hang happens at the point where an "rbd map >> " command is run. >> >> I've attached the ceph logs from the cluster. The map command happens >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can >> be seen in the log as 172.18.0.15:0/1143980479. >> >> It appears as if the TCP socket is opened to the OSD, but then times >> out 15 minutes later, the process gets data when the socket is closed >> on the client server and it retries. >> >> Please help. >> >> We are using ceph version 0.48.2argonaut >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). >> >> We are using a 3.5.7 kernel with the following list of patches applied: >> >> 1-libceph-encapsulate-out-message-data-setup.patch >> 2-libceph-dont-mark-footer-complete-before-it-is.patch >> 3-libceph-move-init-of-bio_iter.patch >> 4-libceph-dont-use-bio_iter-as-a-flag.patch >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch >> 8-libceph-protect-ceph_con_open-with-mutex.patch >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch >> 11-rbd-set-image-size-when-header-is-updated.patch >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch >> 17-libceph-check-for-invalid-mapping.patch >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch >> 19-rbd-BUG-on-invalid-layout.patch >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch >> 21-ceph-avoid-32-bit-page-index-overflow.patch >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch >> >> Any suggestions? > > > The log shows your monitors don't have time sychronized enough among > them to make much progress (including authenticating new connections). > That's probably the real issue. 0.2s is pretty large clock drift. > > >> One thought is that the following patch (which we could not apply) is >> what is required: >> >> 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch > > > This is certainly useful too, but I don't think it's the cause of > the delay in this case. > > Josh > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: changed rbd cp behavior in 0.53
On 11/12/2012 08:30 AM, Andrey Korolyov wrote: Hi, For this version, rbd cp assumes that destination pool is the same as source, not 'rbd', if pool in the destination path is omitted. rbd cp install/img testimg rbd ls install img testimg Is this change permanent? Thanks! This is a regression. The previous behavior will be restored for 0.54. I added http://tracker.newdream.net/issues/3478 to track it. Josh -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On Mon, 12 Nov 2012, Nick Bartos wrote: > After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it > seems we no longer have this hang. Hmm, that's a bit disconcerting. Did this series come from our old 3.5 stable series? I recently prepared a new one that backports *all* of the fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I would be curious if you see problems with that. So far, with these fixes in place, we have not seen any unexplained kernel crashes in this code. I take it you're going back to a 3.5 kernel because you weren't able to get rid of the sync problem with 3.6? sage > > On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin wrote: > > On 11/08/2012 02:10 PM, Mandell Degerness wrote: > >> > >> We are seeing a somewhat random, but frequent hang on our systems > >> during startup. The hang happens at the point where an "rbd map > >> " command is run. > >> > >> I've attached the ceph logs from the cluster. The map command happens > >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can > >> be seen in the log as 172.18.0.15:0/1143980479. > >> > >> It appears as if the TCP socket is opened to the OSD, but then times > >> out 15 minutes later, the process gets data when the socket is closed > >> on the client server and it retries. > >> > >> Please help. > >> > >> We are using ceph version 0.48.2argonaut > >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). > >> > >> We are using a 3.5.7 kernel with the following list of patches applied: > >> > >> 1-libceph-encapsulate-out-message-data-setup.patch > >> 2-libceph-dont-mark-footer-complete-before-it-is.patch > >> 3-libceph-move-init-of-bio_iter.patch > >> 4-libceph-dont-use-bio_iter-as-a-flag.patch > >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch > >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch > >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch > >> 8-libceph-protect-ceph_con_open-with-mutex.patch > >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch > >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch > >> 11-rbd-set-image-size-when-header-is-updated.patch > >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch > >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch > >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch > >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch > >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch > >> 17-libceph-check-for-invalid-mapping.patch > >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch > >> 19-rbd-BUG-on-invalid-layout.patch > >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch > >> 21-ceph-avoid-32-bit-page-index-overflow.patch > >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch > >> > >> Any suggestions? > > > > > > The log shows your monitors don't have time sychronized enough among > > them to make much progress (including authenticating new connections). > > That's probably the real issue. 0.2s is pretty large clock drift. > > > > > >> One thought is that the following patch (which we could not apply) is > >> what is required: > >> > >> 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch > > > > > > This is certainly useful too, but I don't think it's the cause of > > the delay in this case. > > > > Josh > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majord...@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
ceph osd crush set command under 0.53
Did the syntax and behavior of the "ceph osd crush set ..." command change between 0.48 and 0.53? When trying out ceph 0.53, I get the following in my log when trying to add the first OSD to a new cluster (similar behavior for osds 2 and 3). It appears that the ceph osd crush command fails, but still marks the OSDs as up and in: Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 [2012-11-12 23:19:05.759] 908/MainThread savage/INFO: execute(['ceph', 'osd', 'crush', 'set', '0', 'osd.0', '1.0', 'host=172.20.0.13', 'rack=0', 'pool=default']) Nov 12 23:19:05 node-172-20-0-14/172.20.0.14 ceph-mon: 2012-11-12 23:19:05.804080 7ffd761fe700 0 mon.1@1(peon) e1 handle_command mon_command(osd crush set 0 osd.0 1.0 host=172.20.0.13 rack=0 pool=default v 0) v1 Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 23:19:05.772215 7fad40911700 0 mon.0@0(leader) e1 handle_command mon_command(osd crush set 0 osd.0 1.0 host=172.20.0.13 rack=0 pool=default v 0) v1 Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 23:19:05.772248 7fad40911700 0 mon.0@0(leader).osd e2 adding/updating crush item id 0 name 'osd.0' weight 1 at location {host=172.20.0.13,pool=default,rack=0} Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 23:19:05.772323 7fad40911700 1 error: didn't find anywhere to add item 0 in {host=172.20.0.13,pool=default,rack=0} Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 [2012-11-12 23:19:05.783] 908/MainThread savage/CRITICAL: Logging uncaught exception Traceback (most recent call last): File "/usr/bin/sv-fred.py", line 9, in load_entry_point('savage==.2101.118c3ebc8c0843f87e82eb047de043c8a70086bd', 'console_scripts', 'sv-fred.py')() File "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line 811, in main File "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line 798, in run File "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 291, in step File "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 252, in step File "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 231, in _newstate File "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 219, in _newstate File "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line 563, in action_firstboot_full File "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line 768, in handle_message File "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line 750, in start_phase File "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line 164, in start File "/usr/lib64/python2.6/site-packages/savage/utils/__init__.py", line 275, in _wrap File "/usr/lib64/python2.6/site-packages/savage/command/commands/ceph.py", line 50, in crush_myself File "/usr/lib64/python2.6/site-packages/savage/utils/__init__.py", line 244, in execute File "/usr/lib64/python2.6/site-packages/savage/utils/__init__.py", line 130, in collect_subprocess ExecutionError: Command failed: ceph osd crush set 0 osd.0 1.0 host=172.20.0.13 rack=0 pool=default return_code: 1 stdout: (22) Invalid argument stderr: Nov 12 23:19:06 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 23:19:06.491514 7fad40911700 1 mon.0@0(leader).osd e3 e3: 3 osds: 1 up, 1 in Nov 12 23:19:06 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 23:19:06.494461 7fad40911700 0 log [INF] : osdmap e3: 3 osds: 1 up, 1 in Nov 12 23:19:06 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 23:19:06.494463 mon.0 172.20.0.13:6789/0 16 : [INF] osdmap e3: 3 osds: 1 up, 1 in -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ceph osd crush set command under 0.53
On Mon, 12 Nov 2012, Mandell Degerness wrote: > Did the syntax and behavior of the "ceph osd crush set ..." command > change between 0.48 and 0.53? > > When trying out ceph 0.53, I get the following in my log when trying > to add the first OSD to a new cluster (similar behavior for osds 2 and > 3). It appears that the ceph osd crush command fails, but still marks > the OSDs as up and in: The 'pool=default' is changed to 'root=default', as in the root of the crush hierarchy. 'pool' was confusing because there are also rados pools, which are something else entirely. (You can also omit the first '0' (i.e., just 'osd.123' and not [..., '123', 'osd.123', ...]), but both the old and new syntax are supported.) sage > > Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 [2012-11-12 23:19:05.759] > 908/MainThread savage/INFO: execute(['ceph', 'osd', 'crush', 'set', > '0', 'osd.0', '1.0', 'host=172.20.0.13', 'rack=0', 'pool=default']) > Nov 12 23:19:05 node-172-20-0-14/172.20.0.14 ceph-mon: 2012-11-12 > 23:19:05.804080 7ffd761fe700 0 mon.1@1(peon) e1 handle_command > mon_command(osd crush set 0 osd.0 1.0 host=172.20.0.13 rack=0 > pool=default v 0) v1 > Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 > 23:19:05.772215 7fad40911700 0 mon.0@0(leader) e1 handle_command > mon_command(osd crush set 0 osd.0 1.0 host=172.20.0.13 rack=0 > pool=default v 0) v1 > Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 > 23:19:05.772248 7fad40911700 0 mon.0@0(leader).osd e2 adding/updating > crush item id 0 name 'osd.0' weight 1 at location > {host=172.20.0.13,pool=default,rack=0} > Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 > 23:19:05.772323 7fad40911700 1 error: didn't find anywhere to add > item 0 in {host=172.20.0.13,pool=default,rack=0} > Nov 12 23:19:05 node-172-20-0-13/172.20.0.13 [2012-11-12 23:19:05.783] > 908/MainThread savage/CRITICAL: Logging uncaught exception Traceback > (most recent call last): File "/usr/bin/sv-fred.py", line 9, in > > load_entry_point('savage==.2101.118c3ebc8c0843f87e82eb047de043c8a70086bd', > 'console_scripts', 'sv-fred.py')() File > "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line > 811, in main File > "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line > 798, in run File > "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 291, in > step File "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", > line 252, in step File > "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 231, in > _newstate File > "/usr/lib64/python2.6/site-packages/savage/utils/nfa.py", line 219, in > _newstate File > "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line > 563, in action_firstboot_full File > "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line > 768, in handle_message File > "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line > 750, in start_phase File > "/usr/lib64/python2.6/site-packages/savage/services/fred.py", line > 164, in start File > "/usr/lib64/python2.6/site-packages/savage/utils/__init__.py", line > 275, in _wrap File > "/usr/lib64/python2.6/site-packages/savage/command/commands/ceph.py", > line 50, in crush_myself File > "/usr/lib64/python2.6/site-packages/savage/utils/__init__.py", line > 244, in execute File > "/usr/lib64/python2.6/site-packages/savage/utils/__init__.py", line > 130, in collect_subprocess ExecutionError: Command failed: ceph osd > crush set 0 osd.0 1.0 host=172.20.0.13 rack=0 pool=default > return_code: 1 stdout: (22) Invalid argument stderr: > Nov 12 23:19:06 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 > 23:19:06.491514 7fad40911700 1 mon.0@0(leader).osd e3 e3: 3 osds: 1 > up, 1 in > Nov 12 23:19:06 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 > 23:19:06.494461 7fad40911700 0 log [INF] : osdmap e3: 3 osds: 1 up, 1 > in > Nov 12 23:19:06 node-172-20-0-13/172.20.0.13 ceph-mon: 2012-11-12 > 23:19:06.494463 mon.0 172.20.0.13:6789/0 16 : [INF] osdmap e3: 3 osds: > 1 up, 1 in > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Help] Use Ceph RBD as primary storage in CloudStack 4.0
Hi, All Has somebody used Ceph RBD in CloudStack as primary storage? I see that in the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD as primary storage but met with some problems. I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor has to be compiled with RBD enabled .The libvirt version on the Hypervisor has to be at least 0.10 with RBD enabled).Then I download the latest qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them. But when compiling qemu source code, #wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2 #tar jxvf qemu-1.2.0.tar.bz2 # cd qemu-1.2.0 # ./configure --enable-rbd the following errors occur: ERROR: User requested feature rados block device ERROR: configure was not able to find it But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am very confused.How to use Ceph RBD as primary storage in CloudStack on CentOS6.3?Anyone can help me? Best Regards, Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
MDaemon Notification -- Attachment Removed
The following message contained restricted attachment(s) which have been removed: >From : ceph-devel@vger.kernel.org To: libr...@irost.org Subject : Returned mail: Data format error Message-ID: Attachment(s) removed: - instruction.pif -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: improve speed with auth supported=none
On 11/12/2012 01:57 PM, Stefan Priebe wrote: Thanks, this gives another burst for iops. I'm now at 23.000 iops ;-) So for random 4k iops ceph auth and especially the logging is a lot of overhead. How much difference did disabling auth make vs only disabling logging? Josh Greets, Stefan Am 12.11.2012 19:26, schrieb Sébastien Han: I guess you can refer to that link on the list: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/9776 btw do you get 5000 iop/s on the rbd kernel or on a vm disk? cheers. On Mon, Nov 12, 2012 at 4:37 PM, Stefan Priebe - Profihost AG wrote: Hello list, i'm still trying to improve ceph speed. Disable logging on host and rbd client gives me additional 5000 iop/s which is great. But i also wanted to try disabling authentication using: auth supported=none How does this work? Do i just have to place this line under global section in ceph.conf? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: optmize librbd for iops
On 11/12/2012 05:50 AM, Stefan Priebe - Profihost AG wrote: Hello list, are there any plans to optimize librbd for iops? Right now i'm able to get 50.000 iop/s via iscsi and 100.000 iop/s using multipathing with iscsi. With librbd i'm stuck to around 18.000iops. As this scales with more hosts but not with more disks in a vm. It must be limited by rbd implementation in kvm / librbd. It'd be interesting to see which layers are most limiting in this case - qemu/kvm, librados, or librbd. How does rados bench with 4k writes and then 4k reads with many concurrent IOs do? Unfortunately there's no librbd read benchmark yet. Josh -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: improve speed with auth supported=none
Am 13.11.2012 08:42, schrieb Josh Durgin: On 11/12/2012 01:57 PM, Stefan Priebe wrote: Thanks, this gives another burst for iops. I'm now at 23.000 iops ;-) So for random 4k iops ceph auth and especially the logging is a lot of overhead. How much difference did disabling auth make vs only disabling logging? disable debug logging: 3000 iops disable auth logging: 2000 iops Is anybody in the ceph team also interested in a call graph of kvm when VM is doing random 4k write io? Greets Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: optmize librbd for iops
Am 13.11.2012 08:51, schrieb Josh Durgin: On 11/12/2012 05:50 AM, Stefan Priebe - Profihost AG wrote: Hello list, are there any plans to optimize librbd for iops? Right now i'm able to get 50.000 iop/s via iscsi and 100.000 iop/s using multipathing with iscsi. With librbd i'm stuck to around 18.000iops. As this scales with more hosts but not with more disks in a vm. It must be limited by rbd implementation in kvm / librbd. It'd be interesting to see which layers are most limiting in this case - qemu/kvm, librados, or librbd. How does rados bench with 4k writes and then 4k reads with many concurrent IOs do? Right now i'm using qemu-kvm with librbd and fio inside guest. How does the rados bench work? Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html