[ceph-users] Not up file to mount partition of CephFS
Hi all,
When i use ftp up file to mount partition of CephFS, average speed as
normal = 100MB/ s, but recently, the process was interrupted, not continue
to transfer files.
but state ceph still ok:
[root@Ceph-Store-33-2 dd]# ceph -w
health HEALTH_OK
monmap e2: 2 mons at {0=172.30.33.2:6789/0,1=172.30.33.5:6789/0},
election epoch 782, quorum 0,1 0,1
osdmap e4510: 8 osds: 8 up, 8 in
pgmap v337871: 640 pgs: 639 active+clean, 1
active+clean+scrubbing+deep; 20953 GB data, 20989 GB used, 83428 GB / 107
TB avail
mdsmap e1060: 1/1/1 up {0=1=up:active}, 1 up:standby
dmesg:
libceph: wrong peer, want 172.30.33.5:6807/5804, got 172.30.33.5:6807/7258
libceph: osd6 172.30.33.5:6807 wrong peer at address
libceph: wrong peer, want 172.30.33.5:6807/5804, got 172.30.33.5:6807/7258
libceph: osd6 172.30.33.5:6807 wrong peer at address
libceph: wrong peer, want 172.30.33.5:6807/5804, got 172.30.33.5:6807/7258
libceph: osd6 172.30.33.5:6807 wrong peer at address
libceph: wrong peer, want 172.30.33.5:6807/5804, got 172.30.33.5:6807/7258
libceph: osd6 172.30.33.5:6807 wrong peer at address
libceph: tid 13 timed out on osd6, will reset osd
libceph: wrong peer, want 172.30.33.5:6807/5804, got 172.30.33.5:6807/7258
libceph: osd6 172.30.33.5:6807 wrong peer at address
libceph: tid 13 timed out on osd6, will reset osd
libceph: tid 13 timed out on osd6, will reset osd
libceph: wrong peer, want 172.30.33.5:6807/5804, got 172.30.33.5:6807/7258
libceph: osd6 172.30.33.5:6807 wrong peer at address
libceph: tid 13 timed out on osd6, will reset osd
libceph: mon1 172.30.33.5:6789 socket closed
libceph: mon1 172.30.33.5:6789 session lost, hunting for new mon
libceph: mon1 172.30.33.5:6789 connection failed
libceph: tid 13 timed out on osd6, will reset osd
libceph: tid 668 timed out on osd6, will reset osd
libceph: tid 678 timed out on osd6, will reset osd
libceph: tid 678 timed out on osd6, will reset osd
libceph: tid 678 timed out on osd6, will reset osd
libceph: tid 678 timed out on osd6, will reset osd
libceph: tid 678 timed out on osd6, will reset osd
libceph: tid 864 timed out on osd6, will reset osd
libceph: tid 864 timed out on osd6, will reset osd
libceph: tid 864 timed out on osd6, will reset osd
libceph: tid 864 timed out on osd6, will reset osd
libceph: tid 864 timed out on osd6, will reset osd
libceph: tid 864 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
libceph: tid 1206 timed out on osd6, will reset osd
How to fix it?
Thanks all
--
Bui Minh Tien
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Backporting the kernel client
> > I'm probably not the only one who would like to run a > distribution-provided kernel (which for Debian Wheezy/Ubuntu Precise is > 3.2) and still have a recent-enough Ceph kernel client. So I'm wondering > whether it's feasible to backport the kernel client to an earlier kernel. You can grab the 3.8 kernel from debian experimental http://packages.debian.org/search?keywords=linux-image-3.8 I'm using it on a bunch of machines and I know of a few others using it too. > The plan is as follows: > > 1) Grab the Ceph files from https://github.com/ceph/ceph-client (and put > them over the older kernel sources). If I got it right the files are: > include/keys/ceph-type.h include/linux/ceph/* fs/ceph/* net/ceph/* > drivers/block/rbd.c drivers/block/rbd_types.h > > 2) Make (trivial) adjustments to the source code to account for changed > kernel interfaces. > > 3) Compile as modules and install the new Ceph modules under > /lib/modules. > > 4) Reboot to a standard distribution kernel with up-to-date Ceph client. > I would think you should be able to build a dkms package pretty easily, and it would be a lot faster to build than building an entire kernel, and much easier to maintain. Of course that depends on the degree of integration with the kernel... James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
Hi Sam, No problem, I'll leave that debugging turned up high, and do a mkcephfs from scratch and see what happens. Not sure if it will happen again or not. =) Thanks again. - Travis On Mon, Apr 29, 2013 at 5:51 PM, Samuel Just wrote: > Hmm, I need logging from when the corruption happened. If this is > reproducible, can you enable that logging on a clean osd (or better, a > clean cluster) until the assert occurs? > -Sam > > On Mon, Apr 29, 2013 at 2:45 PM, Travis Rhoden wrote: > > Also, I can note that it does not take a full cluster restart to trigger > > this. If I just restart an OSD that was up/in previously, the same error > > can happen (though not every time). So restarting OSD's for me is a bit > > like Russian roullette. =) Even though restarting an OSD may not also > > result in the error, it seems that once it happens that OSD is gone for > > good. No amount of restart has brought any of the dead ones back. > > > > I'd really like to get to the bottom of it. Let me know if I can do > > anything to help. > > > > I may also have to try completely wiping/rebuilding to see if I can make > > this thing usable. > > > > > > On Mon, Apr 29, 2013 at 2:38 PM, Travis Rhoden > wrote: > >> > >> Hi Sam, > >> > >> Thanks for being willing to take a look. > >> > >> I applied the debug settings on one host that 3 out of 3 OSDs with this > >> problem. Then tried to start them up. Here are the resulting logs: > >> > >> https://dl.dropboxusercontent.com/u/23122069/cephlogs.tgz > >> > >> - Travis > >> > >> > >> On Mon, Apr 29, 2013 at 1:04 PM, Samuel Just > wrote: > >>> > >>> You appear to be missing pg metadata for some reason. If you can > >>> reproduce it with > >>> debug osd = 20 > >>> debug filestore = 20 > >>> debug ms = 1 > >>> on all of the OSDs, I should be able to track it down. > >>> > >>> I created a bug: #4855. > >>> > >>> Thanks! > >>> -Sam > >>> > >>> On Mon, Apr 29, 2013 at 9:52 AM, Travis Rhoden > wrote: > >>> > Thanks Greg. > >>> > > >>> > I quit playing with it because every time I restarted the cluster > >>> > (service > >>> > ceph -a restart), I lost more OSDs.. First time it was 1, 2nd 10, > 3rd > >>> > time > >>> > 13... All 13 down OSDs all show the same stacktrace. > >>> > > >>> > - Travis > >>> > > >>> > > >>> > On Mon, Apr 29, 2013 at 11:56 AM, Gregory Farnum > >>> > wrote: > >>> >> > >>> >> This sounds vaguely familiar to me, and I see > >>> >> http://tracker.ceph.com/issues/4052, which is marked as "Can't > >>> >> reproduce" — I think maybe this is fixed in "next" and "master", but > >>> >> I'm not sure. For more than that I'd have to defer to Sage or Sam. > >>> >> -Greg > >>> >> Software Engineer #42 @ http://inktank.com | http://ceph.com > >>> >> > >>> >> > >>> >> On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden > >>> >> wrote: > >>> >> > Hey folks, > >>> >> > > >>> >> > I'm helping put together a new test/experimental cluster, and hit > >>> >> > this > >>> >> > today > >>> >> > when bringing the cluster up for the first time (using mkcephfs). > >>> >> > > >>> >> > After doing the normal "service ceph -a start", I noticed one OSD > >>> >> > was > >>> >> > down, > >>> >> > and a lot of PGs were stuck creating. I tried restarting the down > >>> >> > OSD, > >>> >> > but > >>> >> > it would come up. It always had this error: > >>> >> > > >>> >> > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot > >>> >> > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In > >>> >> > function > >>> >> > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, > hobject_t&, > >>> >> > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 > 18:11:56.399089 > >>> >> > osd/PG.cc: 2556: FAILED assert(values.size() == 1) > >>> >> > > >>> >> > ceph version 0.60-401-g17a3859 > >>> >> > (17a38593d60f5f29b9b66c13c0aaa759762c6d04) > >>> >> > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > >>> >> > ceph::buffer::list*)+0x1ad) [0x2c3c0a] > >>> >> > 2: (OSD::load_pgs()+0x357) [0x28cba0] > >>> >> > 3: (OSD::init()+0x741) [0x290a16] > >>> >> > 4: (main()+0x1427) [0x2155c0] > >>> >> > 5: (__libc_start_main()+0x99) [0xb69bcf42] > >>> >> > NOTE: a copy of the executable, or `objdump -rdS ` is > >>> >> > needed to > >>> >> > interpret this. > >>> >> > > >>> >> > > >>> >> > I then did a full cluster restart, and now I have ten OSDs down -- > >>> >> > each > >>> >> > showing the same exception/failed assert. > >>> >> > > >>> >> > Anybody seen this? > >>> >> > > >>> >> > I know I'm running a weird version -- it's compiled from source, > and > >>> >> > was > >>> >> > provided to me. The OSDs are all on ARM, and the mon is x86_64. > >>> >> > Just > >>> >> > looking to see if anyone has seen this particular stack trace of > >>> >> > load_pgs()/peek_map_epoch() before > >>> >> > > >>> >> > - Travis > >>> >> > > >>> >> > ___ > >>> >> > ceph-users mailing list > >>> >> > ceph-users@lists.ceph.com > >>> >> > http://lists.
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
Hmm, I need logging from when the corruption happened. If this is reproducible, can you enable that logging on a clean osd (or better, a clean cluster) until the assert occurs? -Sam On Mon, Apr 29, 2013 at 2:45 PM, Travis Rhoden wrote: > Also, I can note that it does not take a full cluster restart to trigger > this. If I just restart an OSD that was up/in previously, the same error > can happen (though not every time). So restarting OSD's for me is a bit > like Russian roullette. =) Even though restarting an OSD may not also > result in the error, it seems that once it happens that OSD is gone for > good. No amount of restart has brought any of the dead ones back. > > I'd really like to get to the bottom of it. Let me know if I can do > anything to help. > > I may also have to try completely wiping/rebuilding to see if I can make > this thing usable. > > > On Mon, Apr 29, 2013 at 2:38 PM, Travis Rhoden wrote: >> >> Hi Sam, >> >> Thanks for being willing to take a look. >> >> I applied the debug settings on one host that 3 out of 3 OSDs with this >> problem. Then tried to start them up. Here are the resulting logs: >> >> https://dl.dropboxusercontent.com/u/23122069/cephlogs.tgz >> >> - Travis >> >> >> On Mon, Apr 29, 2013 at 1:04 PM, Samuel Just wrote: >>> >>> You appear to be missing pg metadata for some reason. If you can >>> reproduce it with >>> debug osd = 20 >>> debug filestore = 20 >>> debug ms = 1 >>> on all of the OSDs, I should be able to track it down. >>> >>> I created a bug: #4855. >>> >>> Thanks! >>> -Sam >>> >>> On Mon, Apr 29, 2013 at 9:52 AM, Travis Rhoden wrote: >>> > Thanks Greg. >>> > >>> > I quit playing with it because every time I restarted the cluster >>> > (service >>> > ceph -a restart), I lost more OSDs.. First time it was 1, 2nd 10, 3rd >>> > time >>> > 13... All 13 down OSDs all show the same stacktrace. >>> > >>> > - Travis >>> > >>> > >>> > On Mon, Apr 29, 2013 at 11:56 AM, Gregory Farnum >>> > wrote: >>> >> >>> >> This sounds vaguely familiar to me, and I see >>> >> http://tracker.ceph.com/issues/4052, which is marked as "Can't >>> >> reproduce" — I think maybe this is fixed in "next" and "master", but >>> >> I'm not sure. For more than that I'd have to defer to Sage or Sam. >>> >> -Greg >>> >> Software Engineer #42 @ http://inktank.com | http://ceph.com >>> >> >>> >> >>> >> On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden >>> >> wrote: >>> >> > Hey folks, >>> >> > >>> >> > I'm helping put together a new test/experimental cluster, and hit >>> >> > this >>> >> > today >>> >> > when bringing the cluster up for the first time (using mkcephfs). >>> >> > >>> >> > After doing the normal "service ceph -a start", I noticed one OSD >>> >> > was >>> >> > down, >>> >> > and a lot of PGs were stuck creating. I tried restarting the down >>> >> > OSD, >>> >> > but >>> >> > it would come up. It always had this error: >>> >> > >>> >> > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot >>> >> > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In >>> >> > function >>> >> > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, >>> >> > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 18:11:56.399089 >>> >> > osd/PG.cc: 2556: FAILED assert(values.size() == 1) >>> >> > >>> >> > ceph version 0.60-401-g17a3859 >>> >> > (17a38593d60f5f29b9b66c13c0aaa759762c6d04) >>> >> > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, >>> >> > ceph::buffer::list*)+0x1ad) [0x2c3c0a] >>> >> > 2: (OSD::load_pgs()+0x357) [0x28cba0] >>> >> > 3: (OSD::init()+0x741) [0x290a16] >>> >> > 4: (main()+0x1427) [0x2155c0] >>> >> > 5: (__libc_start_main()+0x99) [0xb69bcf42] >>> >> > NOTE: a copy of the executable, or `objdump -rdS ` is >>> >> > needed to >>> >> > interpret this. >>> >> > >>> >> > >>> >> > I then did a full cluster restart, and now I have ten OSDs down -- >>> >> > each >>> >> > showing the same exception/failed assert. >>> >> > >>> >> > Anybody seen this? >>> >> > >>> >> > I know I'm running a weird version -- it's compiled from source, and >>> >> > was >>> >> > provided to me. The OSDs are all on ARM, and the mon is x86_64. >>> >> > Just >>> >> > looking to see if anyone has seen this particular stack trace of >>> >> > load_pgs()/peek_map_epoch() before >>> >> > >>> >> > - Travis >>> >> > >>> >> > ___ >>> >> > ceph-users mailing list >>> >> > ceph-users@lists.ceph.com >>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> > >>> > >>> > >> >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
Also, I can note that it does not take a full cluster restart to trigger this. If I just restart an OSD that was up/in previously, the same error can happen (though not every time). So restarting OSD's for me is a bit like Russian roullette. =) Even though restarting an OSD may not also result in the error, it seems that once it happens that OSD is gone for good. No amount of restart has brought any of the dead ones back. I'd really like to get to the bottom of it. Let me know if I can do anything to help. I may also have to try completely wiping/rebuilding to see if I can make this thing usable. On Mon, Apr 29, 2013 at 2:38 PM, Travis Rhoden wrote: > Hi Sam, > > Thanks for being willing to take a look. > > I applied the debug settings on one host that 3 out of 3 OSDs with this > problem. Then tried to start them up. Here are the resulting logs: > > https://dl.dropboxusercontent.com/u/23122069/cephlogs.tgz > > - Travis > > > On Mon, Apr 29, 2013 at 1:04 PM, Samuel Just wrote: > >> You appear to be missing pg metadata for some reason. If you can >> reproduce it with >> debug osd = 20 >> debug filestore = 20 >> debug ms = 1 >> on all of the OSDs, I should be able to track it down. >> >> I created a bug: #4855. >> >> Thanks! >> -Sam >> >> On Mon, Apr 29, 2013 at 9:52 AM, Travis Rhoden wrote: >> > Thanks Greg. >> > >> > I quit playing with it because every time I restarted the cluster >> (service >> > ceph -a restart), I lost more OSDs.. First time it was 1, 2nd 10, 3rd >> time >> > 13... All 13 down OSDs all show the same stacktrace. >> > >> > - Travis >> > >> > >> > On Mon, Apr 29, 2013 at 11:56 AM, Gregory Farnum >> wrote: >> >> >> >> This sounds vaguely familiar to me, and I see >> >> http://tracker.ceph.com/issues/4052, which is marked as "Can't >> >> reproduce" — I think maybe this is fixed in "next" and "master", but >> >> I'm not sure. For more than that I'd have to defer to Sage or Sam. >> >> -Greg >> >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> >> >> >> On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden >> wrote: >> >> > Hey folks, >> >> > >> >> > I'm helping put together a new test/experimental cluster, and hit >> this >> >> > today >> >> > when bringing the cluster up for the first time (using mkcephfs). >> >> > >> >> > After doing the normal "service ceph -a start", I noticed one OSD was >> >> > down, >> >> > and a lot of PGs were stuck creating. I tried restarting the down >> OSD, >> >> > but >> >> > it would come up. It always had this error: >> >> > >> >> > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot >> >> > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In function >> >> > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, >> >> > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 18:11:56.399089 >> >> > osd/PG.cc: 2556: FAILED assert(values.size() == 1) >> >> > >> >> > ceph version 0.60-401-g17a3859 >> >> > (17a38593d60f5f29b9b66c13c0aaa759762c6d04) >> >> > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, >> >> > ceph::buffer::list*)+0x1ad) [0x2c3c0a] >> >> > 2: (OSD::load_pgs()+0x357) [0x28cba0] >> >> > 3: (OSD::init()+0x741) [0x290a16] >> >> > 4: (main()+0x1427) [0x2155c0] >> >> > 5: (__libc_start_main()+0x99) [0xb69bcf42] >> >> > NOTE: a copy of the executable, or `objdump -rdS ` is >> >> > needed to >> >> > interpret this. >> >> > >> >> > >> >> > I then did a full cluster restart, and now I have ten OSDs down -- >> each >> >> > showing the same exception/failed assert. >> >> > >> >> > Anybody seen this? >> >> > >> >> > I know I'm running a weird version -- it's compiled from source, and >> was >> >> > provided to me. The OSDs are all on ARM, and the mon is x86_64. >> Just >> >> > looking to see if anyone has seen this particular stack trace of >> >> > load_pgs()/peek_map_epoch() before >> >> > >> >> > - Travis >> >> > >> >> > ___ >> >> > ceph-users mailing list >> >> > ceph-users@lists.ceph.com >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >> > >> > >> > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
Hi Sam, Thanks for being willing to take a look. I applied the debug settings on one host that 3 out of 3 OSDs with this problem. Then tried to start them up. Here are the resulting logs: https://dl.dropboxusercontent.com/u/23122069/cephlogs.tgz - Travis On Mon, Apr 29, 2013 at 1:04 PM, Samuel Just wrote: > You appear to be missing pg metadata for some reason. If you can > reproduce it with > debug osd = 20 > debug filestore = 20 > debug ms = 1 > on all of the OSDs, I should be able to track it down. > > I created a bug: #4855. > > Thanks! > -Sam > > On Mon, Apr 29, 2013 at 9:52 AM, Travis Rhoden wrote: > > Thanks Greg. > > > > I quit playing with it because every time I restarted the cluster > (service > > ceph -a restart), I lost more OSDs.. First time it was 1, 2nd 10, 3rd > time > > 13... All 13 down OSDs all show the same stacktrace. > > > > - Travis > > > > > > On Mon, Apr 29, 2013 at 11:56 AM, Gregory Farnum > wrote: > >> > >> This sounds vaguely familiar to me, and I see > >> http://tracker.ceph.com/issues/4052, which is marked as "Can't > >> reproduce" — I think maybe this is fixed in "next" and "master", but > >> I'm not sure. For more than that I'd have to defer to Sage or Sam. > >> -Greg > >> Software Engineer #42 @ http://inktank.com | http://ceph.com > >> > >> > >> On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden > wrote: > >> > Hey folks, > >> > > >> > I'm helping put together a new test/experimental cluster, and hit this > >> > today > >> > when bringing the cluster up for the first time (using mkcephfs). > >> > > >> > After doing the normal "service ceph -a start", I noticed one OSD was > >> > down, > >> > and a lot of PGs were stuck creating. I tried restarting the down > OSD, > >> > but > >> > it would come up. It always had this error: > >> > > >> > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot > >> > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In function > >> > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > >> > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 18:11:56.399089 > >> > osd/PG.cc: 2556: FAILED assert(values.size() == 1) > >> > > >> > ceph version 0.60-401-g17a3859 > >> > (17a38593d60f5f29b9b66c13c0aaa759762c6d04) > >> > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > >> > ceph::buffer::list*)+0x1ad) [0x2c3c0a] > >> > 2: (OSD::load_pgs()+0x357) [0x28cba0] > >> > 3: (OSD::init()+0x741) [0x290a16] > >> > 4: (main()+0x1427) [0x2155c0] > >> > 5: (__libc_start_main()+0x99) [0xb69bcf42] > >> > NOTE: a copy of the executable, or `objdump -rdS ` is > >> > needed to > >> > interpret this. > >> > > >> > > >> > I then did a full cluster restart, and now I have ten OSDs down -- > each > >> > showing the same exception/failed assert. > >> > > >> > Anybody seen this? > >> > > >> > I know I'm running a weird version -- it's compiled from source, and > was > >> > provided to me. The OSDs are all on ARM, and the mon is x86_64. Just > >> > looking to see if anyone has seen this particular stack trace of > >> > load_pgs()/peek_map_epoch() before > >> > > >> > - Travis > >> > > >> > ___ > >> > ceph-users mailing list > >> > ceph-users@lists.ceph.com > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
You appear to be missing pg metadata for some reason. If you can reproduce it with debug osd = 20 debug filestore = 20 debug ms = 1 on all of the OSDs, I should be able to track it down. I created a bug: #4855. Thanks! -Sam On Mon, Apr 29, 2013 at 9:52 AM, Travis Rhoden wrote: > Thanks Greg. > > I quit playing with it because every time I restarted the cluster (service > ceph -a restart), I lost more OSDs.. First time it was 1, 2nd 10, 3rd time > 13... All 13 down OSDs all show the same stacktrace. > > - Travis > > > On Mon, Apr 29, 2013 at 11:56 AM, Gregory Farnum wrote: >> >> This sounds vaguely familiar to me, and I see >> http://tracker.ceph.com/issues/4052, which is marked as "Can't >> reproduce" — I think maybe this is fixed in "next" and "master", but >> I'm not sure. For more than that I'd have to defer to Sage or Sam. >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden wrote: >> > Hey folks, >> > >> > I'm helping put together a new test/experimental cluster, and hit this >> > today >> > when bringing the cluster up for the first time (using mkcephfs). >> > >> > After doing the normal "service ceph -a start", I noticed one OSD was >> > down, >> > and a lot of PGs were stuck creating. I tried restarting the down OSD, >> > but >> > it would come up. It always had this error: >> > >> > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot >> > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In function >> > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, >> > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 18:11:56.399089 >> > osd/PG.cc: 2556: FAILED assert(values.size() == 1) >> > >> > ceph version 0.60-401-g17a3859 >> > (17a38593d60f5f29b9b66c13c0aaa759762c6d04) >> > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, >> > ceph::buffer::list*)+0x1ad) [0x2c3c0a] >> > 2: (OSD::load_pgs()+0x357) [0x28cba0] >> > 3: (OSD::init()+0x741) [0x290a16] >> > 4: (main()+0x1427) [0x2155c0] >> > 5: (__libc_start_main()+0x99) [0xb69bcf42] >> > NOTE: a copy of the executable, or `objdump -rdS ` is >> > needed to >> > interpret this. >> > >> > >> > I then did a full cluster restart, and now I have ten OSDs down -- each >> > showing the same exception/failed assert. >> > >> > Anybody seen this? >> > >> > I know I'm running a weird version -- it's compiled from source, and was >> > provided to me. The OSDs are all on ARM, and the mon is x86_64. Just >> > looking to see if anyone has seen this particular stack trace of >> > load_pgs()/peek_map_epoch() before >> > >> > - Travis >> > >> > ___ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
Thanks Greg. I quit playing with it because every time I restarted the cluster (service ceph -a restart), I lost more OSDs.. First time it was 1, 2nd 10, 3rd time 13... All 13 down OSDs all show the same stacktrace. - Travis On Mon, Apr 29, 2013 at 11:56 AM, Gregory Farnum wrote: > This sounds vaguely familiar to me, and I see > http://tracker.ceph.com/issues/4052, which is marked as "Can't > reproduce" — I think maybe this is fixed in "next" and "master", but > I'm not sure. For more than that I'd have to defer to Sage or Sam. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden wrote: > > Hey folks, > > > > I'm helping put together a new test/experimental cluster, and hit this > today > > when bringing the cluster up for the first time (using mkcephfs). > > > > After doing the normal "service ceph -a start", I noticed one OSD was > down, > > and a lot of PGs were stuck creating. I tried restarting the down OSD, > but > > it would come up. It always had this error: > > > > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot > > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In function > > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 18:11:56.399089 > > osd/PG.cc: 2556: FAILED assert(values.size() == 1) > > > > ceph version 0.60-401-g17a3859 > (17a38593d60f5f29b9b66c13c0aaa759762c6d04) > > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > > ceph::buffer::list*)+0x1ad) [0x2c3c0a] > > 2: (OSD::load_pgs()+0x357) [0x28cba0] > > 3: (OSD::init()+0x741) [0x290a16] > > 4: (main()+0x1427) [0x2155c0] > > 5: (__libc_start_main()+0x99) [0xb69bcf42] > > NOTE: a copy of the executable, or `objdump -rdS ` is > needed to > > interpret this. > > > > > > I then did a full cluster restart, and now I have ten OSDs down -- each > > showing the same exception/failed assert. > > > > Anybody seen this? > > > > I know I'm running a weird version -- it's compiled from source, and was > > provided to me. The OSDs are all on ARM, and the mon is x86_64. Just > > looking to see if anyone has seen this particular stack trace of > > load_pgs()/peek_map_epoch() before > > > > - Travis > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Failed assert when starting new OSDs in 0.60
This sounds vaguely familiar to me, and I see http://tracker.ceph.com/issues/4052, which is marked as "Can't reproduce" — I think maybe this is fixed in "next" and "master", but I'm not sure. For more than that I'd have to defer to Sage or Sam. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Sat, Apr 27, 2013 at 6:43 PM, Travis Rhoden wrote: > Hey folks, > > I'm helping put together a new test/experimental cluster, and hit this today > when bringing the cluster up for the first time (using mkcephfs). > > After doing the normal "service ceph -a start", I noticed one OSD was down, > and a lot of PGs were stuck creating. I tried restarting the down OSD, but > it would come up. It always had this error: > > -1> 2013-04-27 18:11:56.179804 b6fcd000 2 osd.1 0 boot > 0> 2013-04-27 18:11:56.402161 b6fcd000 -1 osd/PG.cc: In function > 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > ceph::bufferlist*)' thread b6fcd000 time 2013-04-27 18:11:56.399089 > osd/PG.cc: 2556: FAILED assert(values.size() == 1) > > ceph version 0.60-401-g17a3859 (17a38593d60f5f29b9b66c13c0aaa759762c6d04) > 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, > ceph::buffer::list*)+0x1ad) [0x2c3c0a] > 2: (OSD::load_pgs()+0x357) [0x28cba0] > 3: (OSD::init()+0x741) [0x290a16] > 4: (main()+0x1427) [0x2155c0] > 5: (__libc_start_main()+0x99) [0xb69bcf42] > NOTE: a copy of the executable, or `objdump -rdS ` is needed to > interpret this. > > > I then did a full cluster restart, and now I have ten OSDs down -- each > showing the same exception/failed assert. > > Anybody seen this? > > I know I'm running a weird version -- it's compiled from source, and was > provided to me. The OSDs are all on ARM, and the mon is x86_64. Just > looking to see if anyone has seen this particular stack trace of > load_pgs()/peek_map_epoch() before > > - Travis > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Backporting the kernel client
Hi Juha On 29/04/13 15:29, Yehuda Sadeh wrote: On Mon, Apr 29, 2013 at 6:16 AM, Juha Aatrokoski wrote: >I'm probably not the only one who would like to run a distribution-provided >kernel (which for Debian Wheezy/Ubuntu Precise is 3.2) and still have a >recent-enough Ceph kernel client. So I'm wondering whether it's feasible to >backport the kernel client to an earlier kernel. The plan is as follows: Ubuntu 12.04 now includes Hardware Enablement Kernels from Quantal (3.5) and soon from Raring (3.8): http://www.jorgecastro.org/2013/02/19/what-the-lts-enablement-stack-means-for-sysadmins/ This allows you to use a more recent, distro provided kernel than 3.2 which should have the required fixes/features you need. That said if you are having problems with the 3.2 kernel please do report bugs back to Ubuntu. Cheers James -- James Page Technical Lead Ubuntu Server Team james.p...@canonical.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Backporting the kernel client
On Mon, Apr 29, 2013 at 6:16 AM, Juha Aatrokoski wrote: > I'm probably not the only one who would like to run a distribution-provided > kernel (which for Debian Wheezy/Ubuntu Precise is 3.2) and still have a > recent-enough Ceph kernel client. So I'm wondering whether it's feasible to > backport the kernel client to an earlier kernel. The plan is as follows: > > 1) Grab the Ceph files from https://github.com/ceph/ceph-client (and put > them over the older kernel sources). If I got it right the files are: > include/keys/ceph-type.h include/linux/ceph/* fs/ceph/* net/ceph/* > drivers/block/rbd.c drivers/block/rbd_types.h > > 2) Make (trivial) adjustments to the source code to account for changed > kernel interfaces. > > 3) Compile as modules and install the new Ceph modules under /lib/modules. > > 4) Reboot to a standard distribution kernel with up-to-date Ceph client. > > > Now the main questions are: > > Q1: Is the Ceph client contained in the files mentioned in 1), or does it > include changes elsewhere in the kernel that cannot be built as modules? The ceph stuff is mostly there, if you exclude make file changes. Don't think there's anything outside. > > Q2: Regarding 2), are there any nontrivial interface changes i.e. Ceph > actually using newly introduced features instead of just adapting to a > changed syntax? I don't really know, haven't been actively looking at the kernel side for a while. It used to compile on that kernel version, so you can always refer to the older version and see how it used to work. Usually you can just try to compile it, see where it fails and then try to figure out what has changed there. Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Backporting the kernel client
On 04/29/2013 08:16 AM, Juha Aatrokoski wrote: > I'm probably not the only one who would like to run a > distribution-provided kernel (which for Debian Wheezy/Ubuntu Precise is > 3.2) and still have a recent-enough Ceph kernel client. So I'm wondering > whether it's feasible to backport the kernel client to an earlier > kernel. The plan is as follows: > > 1) Grab the Ceph files from https://github.com/ceph/ceph-client (and put > them over the older kernel sources). If I got it right the files are: > include/keys/ceph-type.h include/linux/ceph/* fs/ceph/* net/ceph/* > drivers/block/rbd.c drivers/block/rbd_types.h That is the correct and complete list of source files. (I didn't know about include/keys/ceph-type.h.) > 2) Make (trivial) adjustments to the source code to account for changed > kernel interfaces. > > 3) Compile as modules and install the new Ceph modules under /lib/modules. > 4) Reboot to a standard distribution kernel with up-to-date Ceph client. That's roughly correct. There may be some little details (like running "depmod") but you've got it. > Now the main questions are: > > Q1: Is the Ceph client contained in the files mentioned in 1), or does > it include changes elsewhere in the kernel that cannot be built as modules? They are as you described above. > Q2: Regarding 2), are there any nontrivial interface changes i.e. Ceph > actually using newly introduced features instead of just adapting to a > changed syntax? I'm not aware of any, but I haven't tried a particular port. > Any other thoughts or comments on this? Let me know if you try it and run into trouble, I may be able to help. -Alex > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Stuck pages and other bad things
I'm presuming this is the correct list (rather than the -devel list) please correct me if I'm wrong there. I setup ceph (0.56.4) a few months ago with two disk servers and one dedicated monitor. The disk servers also have monitors, so there are a total of 3 monitors for the cluster. Each of the disk servers has 8 OSDs. I didn't actually get a 'ceph osd tree' output from that, but cutting-and-pasting relevant parts from what I have now, it probably looked like this: # id weight type name up/down reweight -1 16 root default -3 16 rack unknownrack -2 0 host leviathan 100 1 osd.100 up 1 101 1 osd.101 up 1 102 1 osd.102 up 1 103 1 osd.103 up 1 104 1 osd.104 up 1 105 1 osd.105 up 1 106 1 osd.106 up 1 107 1 osd.107 up 1 -4 8 host minotaur 200 1 osd.200 up 1 201 1 osd.201 up 1 202 1 osd.202 up 1 203 1 osd.203 up 1 204 1 osd.204 up 1 205 1 osd.205 up 1 206 1 osd.206 up 1 207 1 osd.207 up 1 A couple of weeks ago, for valid reasons that aren't relevant here, we decided to repurpose one of the disk servers (leviathan) and replace the ceph fileserver with some other hardware. I created a new server (aergia). That changed the 'ceph osd tree' to this: # id weight type name up/down reweight -1 16 root default -3 16 rack unknownrack -2 0 host leviathan 100 1 osd.100 up 1 101 1 osd.101 up 1 102 1 osd.102 up 1 103 1 osd.103 up 1 104 1 osd.104 up 1 105 1 osd.105 up 1 106 1 osd.106 up 1 107 1 osd.107 up 1 -4 8 host minotaur 200 1 osd.200 up 1 201 1 osd.201 up 1 202 1 osd.202 up 1 203 1 osd.203 up 1 204 1 osd.204 up 1 205 1 osd.205 up 1 206 1 osd.206 up 1 207 1 osd.207 up 1 0 1 osd.0 up 1 1 1 osd.1 up 1 2 1 osd.2 up 1 3 1 osd.3 up 1 4 1 osd.4 up 1 5 1 osd.5 up 1 6 1 osd.6 up 1 7 1 osd.7 up 1 Everything was looking happy, so I began removing the OSDs on leviathan. That's when the problems stared. 'ceph health detail' shows that there are several pages that either existed only on that disk server, e.g. pg 0.312 is stuck unclean since forever, current state stale+active+degraded+remapped, last acting [103] or pages that were only replicated back onto the same host, e.g. pg 0.2f4 is stuck unclean since forever, current state stale+active+remapped, last acting [106,101] I brought leviathan back up, and I *think* everything is at least responding now. But 'ceph health' still shows HEALTH_WARN 302 pgs degraded; 810 pgs stale; 810 pgs stuck stale; 3562 pgs stuck unclean; recovery 44951/2289634 degraded (1.963%) ...and it's been stuck there for a long time. So my question is, how do I force data off the to-be-decommissioned server safely and get back to "HEALTH_OK"? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Backporting the kernel client
I'm probably not the only one who would like to run a distribution-provided kernel (which for Debian Wheezy/Ubuntu Precise is 3.2) and still have a recent-enough Ceph kernel client. So I'm wondering whether it's feasible to backport the kernel client to an earlier kernel. The plan is as follows: 1) Grab the Ceph files from https://github.com/ceph/ceph-client (and put them over the older kernel sources). If I got it right the files are: include/keys/ceph-type.h include/linux/ceph/* fs/ceph/* net/ceph/* drivers/block/rbd.c drivers/block/rbd_types.h 2) Make (trivial) adjustments to the source code to account for changed kernel interfaces. 3) Compile as modules and install the new Ceph modules under /lib/modules. 4) Reboot to a standard distribution kernel with up-to-date Ceph client. Now the main questions are: Q1: Is the Ceph client contained in the files mentioned in 1), or does it include changes elsewhere in the kernel that cannot be built as modules? Q2: Regarding 2), are there any nontrivial interface changes i.e. Ceph actually using newly introduced features instead of just adapting to a changed syntax? Any other thoughts or comments on this? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] min_size write behavior
Reading about min_size behavior, I understand next: if I set min_size=1 - it is
for reading & writing, so it is dangerous. Can I (via crush map) or developers
(via code changes) to separate min_size behavior for reading & writing? I event
trying to look into code, but IMHO developers can solve all more easy.
Thinking about this, I prefer to see next behaviour (for example): on write, if
PG number < size, but >= min_size - before writing heal this PG to consistent
state.
It can contains from this steps (I don't understand now code structure, but have
probably ideas about tasks separating):
1) On repair (healing) proceed write pending PGs first (this is separated good
idea - IMHO) - to minimize client freezing;
2) Minimize period for this write pending PGs before healing (same);
3) Make min_size [optional] working only for read.
Or simple always clone ("heal") inconsistent PGs to "size" in write task (if
code enables it).
So write requests will be always protected from data loss (of course, still
possibility to invert written and offline OSDs in one pass in large cluster, but
this is minimal care for mind about min_size).
--
WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.unibel.by/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
