Re: [ceph-users] OSD won't go up after node reboot
Data lives in another container attached to OSD container as Docker volume. According to `deis ps -a`, this volume was created two weeks ago, though all files in `current` are very recent. I suspect that something removed files in the data volume after reboot. As reboot was caused by CoreOS update, it might be newer version of Docker (1.6 -> 1.7) that introduced the problem. Or maybe it was container initialization process that somehow removed and recreated files. I don't have this data volume anymore, so can only guess. 2015-08-31 18:28 GMT+03:00 Jan Schermer <j...@schermer.cz>: > Is it possible that something else was mounted there? > Or is it possible nothing was mounted there? > That would explain such behaviour... > > Jan > > On 31 Aug 2015, at 17:07, Евгений Д. <ineu.m...@gmail.com> wrote: > > No, it really was in the cluster. Before reboot cluster had HEALTH_OK. > Though now I've checked `current` directory and it doesn't contain any > data: > > root@staging-coreos-1:/var/lib/ceph/osd/ceph-0# ls current > commit_op_seq meta nosnap omap > > while other OSDs do. It really looks like something was broken on reboot, > probably during container start, so it's not really related to Ceph. I'll > go with OSD recreation. > > Thank you. > > 2015-08-31 11:50 GMT+03:00 Gregory Farnum <gfar...@redhat.com>: > >> On Sat, Aug 29, 2015 at 3:32 PM, Евгений Д. <ineu.m...@gmail.com> wrote: >> > I'm running 3-node cluster with Ceph (it's Deis cluster, so Ceph >> daemons are >> > containerized). There are 3 OSDs and 3 mons. After rebooting all nodes >> one >> > by one all monitors are up, but only two OSDs of three are up. 'Down' >> OSD is >> > really running but is never marked up/in. >> > All three mons are reachable from inside the OSD container. >> > I've run `log dump` for this OSD and found this line: >> > >> > Aug 29 06:19:39 staging-coreos-1 sh[7393]: -99> 2015-08-29 >> 06:18:51.855432 >> > 7f5902009700 3 osd.0 0 handle_osd_map epochs [1,90], i have 0, src has >> > [1,90] >> > >> > Is it the reason why OSD cannot connect to the cluster? If yes, why >> could it >> > happen? I haven't removed any data from /var/lib/ceph/osd. >> > Is it possible to bring this OSD back to cluster without completely >> > recreating it? >> > >> > Ceph version is: >> > >> > root@staging-coreos-1:/# ceph -v >> > ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) >> >> It's pretty unlikely. I presume (since the OSD has no maps) that it's >> never actually been up and in the cluster? Or else its data store has >> been pretty badly corrupted since it doesn't have any of the requisite >> metadata. In which case you'll probably be best off recreating it >> (with 3 OSDs I assume all your PGs are still active). >> -Greg >> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD won't go up after node reboot
No, it really was in the cluster. Before reboot cluster had HEALTH_OK. Though now I've checked `current` directory and it doesn't contain any data: root@staging-coreos-1:/var/lib/ceph/osd/ceph-0# ls current commit_op_seq meta nosnap omap while other OSDs do. It really looks like something was broken on reboot, probably during container start, so it's not really related to Ceph. I'll go with OSD recreation. Thank you. 2015-08-31 11:50 GMT+03:00 Gregory Farnum <gfar...@redhat.com>: > On Sat, Aug 29, 2015 at 3:32 PM, Евгений Д. <ineu.m...@gmail.com> wrote: > > I'm running 3-node cluster with Ceph (it's Deis cluster, so Ceph daemons > are > > containerized). There are 3 OSDs and 3 mons. After rebooting all nodes > one > > by one all monitors are up, but only two OSDs of three are up. 'Down' > OSD is > > really running but is never marked up/in. > > All three mons are reachable from inside the OSD container. > > I've run `log dump` for this OSD and found this line: > > > > Aug 29 06:19:39 staging-coreos-1 sh[7393]: -99> 2015-08-29 > 06:18:51.855432 > > 7f5902009700 3 osd.0 0 handle_osd_map epochs [1,90], i have 0, src has > > [1,90] > > > > Is it the reason why OSD cannot connect to the cluster? If yes, why > could it > > happen? I haven't removed any data from /var/lib/ceph/osd. > > Is it possible to bring this OSD back to cluster without completely > > recreating it? > > > > Ceph version is: > > > > root@staging-coreos-1:/# ceph -v > > ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) > > It's pretty unlikely. I presume (since the OSD has no maps) that it's > never actually been up and in the cluster? Or else its data store has > been pretty badly corrupted since it doesn't have any of the requisite > metadata. In which case you'll probably be best off recreating it > (with 3 OSDs I assume all your PGs are still active). > -Greg > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSD won't go up after node reboot
I'm running 3-node cluster with Ceph (it's Deis cluster, so Ceph daemons are containerized). There are 3 OSDs and 3 mons. After rebooting all nodes one by one all monitors are up, but only two OSDs of three are up. 'Down' OSD is really running but is never marked up/in. All three mons are reachable from inside the OSD container. I've run `log dump` for this OSD and found this line: Aug 29 06:19:39 staging-coreos-1 sh[7393]: -99 2015-08-29 06:18:51.855432 7f5902009700 3 osd.0 0 handle_osd_map epochs [1,90], i have 0, src has [1,90] Is it the reason why OSD cannot connect to the cluster? If yes, why could it happen? I haven't removed any data from /var/lib/ceph/osd. Is it possible to bring this OSD back to cluster without completely recreating it? Ceph version is: root@staging-coreos-1:/# ceph -v ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-osd suddenly dies and no longer can be started
I kind of fixed it by creating a new journal in file instead of separate partition, which probably caused some data loss, but at least allowed OSD to start and join cluster. Backfilling is now in progress. Old journal is still there on separate device, if it can help in investigation. But this malloc - ENOMEM/OOM killer - corrupted journal - trying to recover - ENOMEM/OOM killer ... looks like a bug. 2015-08-19 0:13 GMT+03:00 Евгений Д. ineu.m...@gmail.com: Hello. I have a small Ceph cluster running 9 OSDs, using XFS on separate disks and dedicated partitions on system disk for journals. After creation it worked fine for a while. Then suddenly one of OSDs stopped and didn't start. I had to recreate it. Recovery started. After few days of recovery OSD on another machine also stopped. I try to start it, it runs for few minutes and dies, looks like it is not able to recover journal. According to strace, it tries to allocate too much memory and stops with ENOMEM. Sometimes it is being killed by kernel's OOM killer. I tried flushing journal manually with `ceph-osd -i 3 --flush-journal`, but it didn't work either. Error log is as follows: [root@assets-2 ~]# ceph-osd -i 3 --flush-journal SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0d 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2015-08-18 23:00:37.956714 7ff102040880 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find 225eff8c/default.4323.18_22783306dc51892b40b49e3e26f79baf_55c38b33172600566c46_s.jpeg/head//8 in index: (2) No such file or directory 2015-08-18 23:00:37.956741 7ff102040880 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find 235eff8c/default.4323.16_3018ff7c6066bddc0c867b293724d7b1_dolar7_106_m.jpg/head//8 in index: (2) No such file or directory skipped 2015-08-18 23:00:37.958424 7ff102040880 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find c//head//8 in index: (2) No such file or directory tcmalloc: large alloc 1073741824 bytes == 0x66b1 @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) tcmalloc: large alloc 2147483648 bytes == 0xbf49 @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) tcmalloc: large alloc 4294967296 bytes == 0x16e32 @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) tcmalloc: large alloc 8589934592 bytes == (nil) @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc *** Caught signal (Aborted) ** in thread 7ff102040880 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: ceph-osd() [0xac5642] 2: (()+0xf130) [0x7ff1009d4130] 3: (gsignal()+0x37) [0x7ff0ff3ee5d7] 4: (abort()+0x148) [0x7ff0ff3efcc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7ff0ffcf29b5] 6: (()+0x5e926) [0x7ff0ffcf0926] 7: (()+0x5e953) [0x7ff0ffcf0953] 8: (()+0x5eb73) [0x7ff0ffcf0b73] 9: (()+0x15d3e) [0x7ff10115ad3e] 10: (tc_new()+0x1e0) [0x7ff10117ade0] 11: (std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocatorchar const)+0x59) [0x7ff0ffd4fc29] 12: (std::string::_Rep::_M_clone(std::allocatorchar const, unsigned long)+0x1b) [0x7ff0ffd5086b] 13: (std::string::reserve(unsigned long)+0x44) [0x7ff0ffd50914] 14: (std::string::append(char const*, unsigned long)+0x4f) [0x7ff0ffd50b7f] 15: (LevelDBStore::LevelDBTransactionImpl::rmkeys_by_prefix(std::string const)+0xdf) [0x968a0f] 16: (DBObjectMap::clear_header(std::tr1::shared_ptrDBObjectMap::_Header, std::tr1::shared_ptrKeyValueDB::TransactionImpl)+0xd3) [0xa572b3] 17: (DBObjectMap::_clear(std::tr1::shared_ptrDBObjectMap::_Header, std::tr1::shared_ptrKeyValueDB::TransactionImpl)+0xa1) [0xa5c6b1] 18: (DBObjectMap::clear(ghobject_t const, SequencerPosition const*)+0x202) [0xa5f762] 19: (FileStore::lfn_unlink(coll_t, ghobject_t const, SequencerPosition const, bool)+0x16a) [0x9018ba] 20: (FileStore::_remove(coll_t, ghobject_t const, SequencerPosition const)+0x9e) [0x90238e] 21: (FileStore::_do_transaction(ObjectStore::Transaction, unsigned long, int, ThreadPool::TPHandle*)+0x252c) [0x911b2c] 22: (FileStore::_do_transactions(std::listObjectStore::Transaction*, std::allocatorObjectStore::Transaction* , unsigned long, ThreadPool::TPHandle*)+0x64
[ceph-users] ceph-osd suddenly dies and no longer can be started
Hello. I have a small Ceph cluster running 9 OSDs, using XFS on separate disks and dedicated partitions on system disk for journals. After creation it worked fine for a while. Then suddenly one of OSDs stopped and didn't start. I had to recreate it. Recovery started. After few days of recovery OSD on another machine also stopped. I try to start it, it runs for few minutes and dies, looks like it is not able to recover journal. According to strace, it tries to allocate too much memory and stops with ENOMEM. Sometimes it is being killed by kernel's OOM killer. I tried flushing journal manually with `ceph-osd -i 3 --flush-journal`, but it didn't work either. Error log is as follows: [root@assets-2 ~]# ceph-osd -i 3 --flush-journal SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0d 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2015-08-18 23:00:37.956714 7ff102040880 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find 225eff8c/default.4323.18_22783306dc51892b40b49e3e26f79baf_55c38b33172600566c46_s.jpeg/head//8 in index: (2) No such file or directory 2015-08-18 23:00:37.956741 7ff102040880 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find 235eff8c/default.4323.16_3018ff7c6066bddc0c867b293724d7b1_dolar7_106_m.jpg/head//8 in index: (2) No such file or directory skipped 2015-08-18 23:00:37.958424 7ff102040880 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find c//head//8 in index: (2) No such file or directory tcmalloc: large alloc 1073741824 bytes == 0x66b1 @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) tcmalloc: large alloc 2147483648 bytes == 0xbf49 @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) tcmalloc: large alloc 4294967296 bytes == 0x16e32 @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) tcmalloc: large alloc 8589934592 bytes == (nil) @ 0x7ff10115ae6a 0x7ff10117ad64 0x7ff0ffd4fc29 0x7ff0ffd5086b 0x7ff0ffd50914 0x7ff0ffd50b7f 0x968a0f 0xa572b3 0xa5c6b1 0xa5f762 0x9018ba 0x90238e 0x911b2c 0x915064 0x92d7cb 0x8ff890 0x642239 0x7ff0ff3daaf5 0x65cdc9 (nil) terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc *** Caught signal (Aborted) ** in thread 7ff102040880 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: ceph-osd() [0xac5642] 2: (()+0xf130) [0x7ff1009d4130] 3: (gsignal()+0x37) [0x7ff0ff3ee5d7] 4: (abort()+0x148) [0x7ff0ff3efcc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7ff0ffcf29b5] 6: (()+0x5e926) [0x7ff0ffcf0926] 7: (()+0x5e953) [0x7ff0ffcf0953] 8: (()+0x5eb73) [0x7ff0ffcf0b73] 9: (()+0x15d3e) [0x7ff10115ad3e] 10: (tc_new()+0x1e0) [0x7ff10117ade0] 11: (std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocatorchar const)+0x59) [0x7ff0ffd4fc29] 12: (std::string::_Rep::_M_clone(std::allocatorchar const, unsigned long)+0x1b) [0x7ff0ffd5086b] 13: (std::string::reserve(unsigned long)+0x44) [0x7ff0ffd50914] 14: (std::string::append(char const*, unsigned long)+0x4f) [0x7ff0ffd50b7f] 15: (LevelDBStore::LevelDBTransactionImpl::rmkeys_by_prefix(std::string const)+0xdf) [0x968a0f] 16: (DBObjectMap::clear_header(std::tr1::shared_ptrDBObjectMap::_Header, std::tr1::shared_ptrKeyValueDB::TransactionImpl)+0xd3) [0xa572b3] 17: (DBObjectMap::_clear(std::tr1::shared_ptrDBObjectMap::_Header, std::tr1::shared_ptrKeyValueDB::TransactionImpl)+0xa1) [0xa5c6b1] 18: (DBObjectMap::clear(ghobject_t const, SequencerPosition const*)+0x202) [0xa5f762] 19: (FileStore::lfn_unlink(coll_t, ghobject_t const, SequencerPosition const, bool)+0x16a) [0x9018ba] 20: (FileStore::_remove(coll_t, ghobject_t const, SequencerPosition const)+0x9e) [0x90238e] 21: (FileStore::_do_transaction(ObjectStore::Transaction, unsigned long, int, ThreadPool::TPHandle*)+0x252c) [0x911b2c] 22: (FileStore::_do_transactions(std::listObjectStore::Transaction*, std::allocatorObjectStore::Transaction* , unsigned long, ThreadPool::TPHandle*)+0x64) [0x915064] 23: (JournalingObjectStore::journal_replay(unsigned long)+0x5db) [0x92d7cb] 24: (FileStore::mount()+0x3730) [0x8ff890] 25: (main()+0xec9) [0x642239] 26: (__libc_start_main()+0xf5) [0x7ff0ff3daaf5] 27: ceph-osd() [0x65cdc9] 2015-08-18 23:02:38.167194 7ff102040880 -1 *** Caught signal (Aborted) ** in thread 7ff102040880 I can recreate filesystem on this OSD's disk and recreate OSD, but I'm not sure that this won't happen with another OSD on this or another machine, and eventually I won't lose all my data because it doesn't
