Hello,
I'm (almost) a new user of ceph (couple of month). In my university, we start to
do some test with ceph a couple of months ago.
We have 2 clusters. Each cluster have 100 OSDs on 10 servers :
Each server as this setup :
CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Memory : 128GB of Memory
OS Storage : 2 x SSD 240GB Intel S3500 DC (raid 1)
Journal Storage : 2 x SSD 400GB Intel S3300 DC (no Raid)
OSD Disk : 10 x HGST ultrastar-7k6000 6TB
Network : 1 x 10Gb/s
OS : Ubuntu 14.04
Ceph version : infernalis 9.2.0
One cluster give access to some user through a S3 gateway (service is still in
beta). We call this cluster "ceph-beta".
One cluster is for our internal need to learn more about ceph. We call this
cluster "ceph-test". (those servers will be integrated into the ceph-beta
cluster when we will need more space)
We have deploy both clusters with the ceph-ansible playbook[1]
Journal are raw partitions on SSDs (400GB Intel S3300 DC) with no raid. 5
journals partitions on each SSDs.
OSDs disk are format in XFS.
1. https://github.com/ceph/ceph-ansible
We have an issue. Some OSDs go down and don't start. It seem to be related to
the fsid of the journal partition :
> -1> 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open:
> ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected
> eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
in attachment, the full logs of one of the dead OSDs
We had this issue with 2 OSDs on ceph-beta cluster fixed by removing, zapping
and readding it.
Now, we have the same issue on ceph-test cluster but on 18 OSDs.
Now the stats of this cluster
> root@icadmin004:~# ceph -s
> cluster 4fb4773c-0873-44ad-a65f-269f01bfcff8
> health HEALTH_WARN
> 1024 pgs incomplete
> 1024 pgs stuck inactive
> 1024 pgs stuck unclean
> monmap e1: 3 mons at
> {iccluster003=10.90.37.4:6789/0,iccluster014=10.90.37.15:6789/0,iccluster022=10.90.37.23:6789/0}
> election epoch 62, quorum 0,1,2
> iccluster003,iccluster014,iccluster022
> osdmap e242: 100 osds: 82 up, 82 in
> flags sortbitwise
> pgmap v469212: 2304 pgs, 10 pools, 2206 bytes data, 181 objects
> 4812 MB used, 447 TB / 447 TB avail
> 1280 active+clean
> 1024 creating+incomplete
We have install this cluster at the begin of February. We did not use that
cluster at all even at the begin to troubleshoot an issue with ceph-ansible. We
did not push any data neither create pool. What could explain this behaviour ?
Thanks for your help
Best regards,
--
Yoann Moulin
EPFL IC-IT
2016-03-03 14:09:00.433074 7efd1a9d5940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4446
2016-03-03 14:09:01.315583 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:01.338328 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:01.338335 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:01.338362 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:01.341468 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:01.341517 7efd1a9d5940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:01.411145 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:01.692400 7efd1a9d5940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:01.694251 7efd1a9d5940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7efd1a9d5940 time 2016-03-03 14:09:01.692413
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7efd1a4cbf2b]
2: (()+0x2c2f80) [0x7efd19ed1f80]
3: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
5: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
6: (OSD::init()+0x26d) [0x7efd19f51a5d]
7: (main()+0x2954) [0x7efd19ed7474]
8: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
9: (()+0x2f82b7) [0x7efd19f072b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-24> 2016-03-03 14:09:00.430918 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perfcounters_dump hook 0x7efd1e3e0050
-23> 2016-03-03 14:09:00.430937 7efd1a9d5940 5 asok(0x7efd1e460000) register_command 1 hook 0x7efd1e3e0050
-22> 2016-03-03 14:09:00.430938 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perf dump hook 0x7efd1e3e0050
-21> 2016-03-03 14:09:00.430940 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perfcounters_schema hook 0x7efd1e3e0050
-20> 2016-03-03 14:09:00.430941 7efd1a9d5940 5 asok(0x7efd1e460000) register_command 2 hook 0x7efd1e3e0050
-19> 2016-03-03 14:09:00.430943 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perf schema hook 0x7efd1e3e0050
-18> 2016-03-03 14:09:00.430945 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perf reset hook 0x7efd1e3e0050
-17> 2016-03-03 14:09:00.430947 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config show hook 0x7efd1e3e0050
-16> 2016-03-03 14:09:00.430950 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config set hook 0x7efd1e3e0050
-15> 2016-03-03 14:09:00.430954 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config get hook 0x7efd1e3e0050
-14> 2016-03-03 14:09:00.430955 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config diff hook 0x7efd1e3e0050
-13> 2016-03-03 14:09:00.430957 7efd1a9d5940 5 asok(0x7efd1e460000) register_command log flush hook 0x7efd1e3e0050
-12> 2016-03-03 14:09:00.430958 7efd1a9d5940 5 asok(0x7efd1e460000) register_command log dump hook 0x7efd1e3e0050
-11> 2016-03-03 14:09:00.430959 7efd1a9d5940 5 asok(0x7efd1e460000) register_command log reopen hook 0x7efd1e3e0050
-10> 2016-03-03 14:09:00.433016 7efd1a9d5940 0 set uid:gid to 64045:64045
-9> 2016-03-03 14:09:00.433074 7efd1a9d5940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4446
-8> 2016-03-03 14:09:01.315583 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
-7> 2016-03-03 14:09:01.338328 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-6> 2016-03-03 14:09:01.338335 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-5> 2016-03-03 14:09:01.338362 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
-4> 2016-03-03 14:09:01.341468 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-3> 2016-03-03 14:09:01.341517 7efd1a9d5940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
-2> 2016-03-03 14:09:01.411145 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-1> 2016-03-03 14:09:01.692400 7efd1a9d5940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
0> 2016-03-03 14:09:01.694251 7efd1a9d5940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7efd1a9d5940 time 2016-03-03 14:09:01.692413
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7efd1a4cbf2b]
2: (()+0x2c2f80) [0x7efd19ed1f80]
3: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
5: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
6: (OSD::init()+0x26d) [0x7efd19f51a5d]
7: (main()+0x2954) [0x7efd19ed7474]
8: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
9: (()+0x2f82b7) [0x7efd19f072b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:01.696632 7efd1a9d5940 -1 *** Caught signal (Aborted) **
in thread 7efd1a9d5940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7efd1a3df2ca]
2: (()+0x10340) [0x7efd18af3340]
3: (gsignal()+0x39) [0x7efd16d6ecc9]
4: (abort()+0x148) [0x7efd16d720d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd17679535]
6: (()+0x5e6d6) [0x7efd176776d6]
7: (()+0x5d799) [0x7efd17676799]
8: (__gxx_personality_v0()+0x52a) [0x7efd1767734a]
9: (()+0xfff3) [0x7efd1710cff3]
10: (_Unwind_Resume()+0x57) [0x7efd1710d517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7efd1a4cc136]
12: (()+0x2c2f80) [0x7efd19ed1f80]
13: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
15: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
16: (OSD::init()+0x26d) [0x7efd19f51a5d]
17: (main()+0x2954) [0x7efd19ed7474]
18: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
19: (()+0x2f82b7) [0x7efd19f072b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-03-03 14:09:01.696632 7efd1a9d5940 -1 *** Caught signal (Aborted) **
in thread 7efd1a9d5940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7efd1a3df2ca]
2: (()+0x10340) [0x7efd18af3340]
3: (gsignal()+0x39) [0x7efd16d6ecc9]
4: (abort()+0x148) [0x7efd16d720d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd17679535]
6: (()+0x5e6d6) [0x7efd176776d6]
7: (()+0x5d799) [0x7efd17676799]
8: (__gxx_personality_v0()+0x52a) [0x7efd1767734a]
9: (()+0xfff3) [0x7efd1710cff3]
10: (_Unwind_Resume()+0x57) [0x7efd1710d517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7efd1a4cc136]
12: (()+0x2c2f80) [0x7efd19ed1f80]
13: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
15: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
16: (OSD::init()+0x26d) [0x7efd19f51a5d]
17: (main()+0x2954) [0x7efd19ed7474]
18: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
19: (()+0x2f82b7) [0x7efd19f072b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:01.889329 7ff0a521d940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4750
2016-03-03 14:09:01.897263 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:01.897561 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:01.897565 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:01.897576 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:01.918907 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:01.918937 7ff0a521d940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:01.965269 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:01.966401 7ff0a521d940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:01.968379 7ff0a521d940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ff0a521d940 time 2016-03-03 14:09:01.966412
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff0a4d13f2b]
2: (()+0x2c2f80) [0x7ff0a4719f80]
3: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
5: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
6: (OSD::init()+0x26d) [0x7ff0a4799a5d]
7: (main()+0x2954) [0x7ff0a471f474]
8: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
9: (()+0x2f82b7) [0x7ff0a474f2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-24> 2016-03-03 14:09:01.887108 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perfcounters_dump hook 0x7ff0a7caa050
-23> 2016-03-03 14:09:01.887136 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command 1 hook 0x7ff0a7caa050
-22> 2016-03-03 14:09:01.887139 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perf dump hook 0x7ff0a7caa050
-21> 2016-03-03 14:09:01.887141 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perfcounters_schema hook 0x7ff0a7caa050
-20> 2016-03-03 14:09:01.887142 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command 2 hook 0x7ff0a7caa050
-19> 2016-03-03 14:09:01.887144 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perf schema hook 0x7ff0a7caa050
-18> 2016-03-03 14:09:01.887147 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perf reset hook 0x7ff0a7caa050
-17> 2016-03-03 14:09:01.887148 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config show hook 0x7ff0a7caa050
-16> 2016-03-03 14:09:01.887150 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config set hook 0x7ff0a7caa050
-15> 2016-03-03 14:09:01.887153 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config get hook 0x7ff0a7caa050
-14> 2016-03-03 14:09:01.887154 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config diff hook 0x7ff0a7caa050
-13> 2016-03-03 14:09:01.887159 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command log flush hook 0x7ff0a7caa050
-12> 2016-03-03 14:09:01.887160 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command log dump hook 0x7ff0a7caa050
-11> 2016-03-03 14:09:01.887161 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command log reopen hook 0x7ff0a7caa050
-10> 2016-03-03 14:09:01.889275 7ff0a521d940 0 set uid:gid to 64045:64045
-9> 2016-03-03 14:09:01.889329 7ff0a521d940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4750
-8> 2016-03-03 14:09:01.897263 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
-7> 2016-03-03 14:09:01.897561 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-6> 2016-03-03 14:09:01.897565 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-5> 2016-03-03 14:09:01.897576 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
-4> 2016-03-03 14:09:01.918907 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-3> 2016-03-03 14:09:01.918937 7ff0a521d940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
-2> 2016-03-03 14:09:01.965269 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-1> 2016-03-03 14:09:01.966401 7ff0a521d940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
0> 2016-03-03 14:09:01.968379 7ff0a521d940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ff0a521d940 time 2016-03-03 14:09:01.966412
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff0a4d13f2b]
2: (()+0x2c2f80) [0x7ff0a4719f80]
3: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
5: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
6: (OSD::init()+0x26d) [0x7ff0a4799a5d]
7: (main()+0x2954) [0x7ff0a471f474]
8: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
9: (()+0x2f82b7) [0x7ff0a474f2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:01.970874 7ff0a521d940 -1 *** Caught signal (Aborted) **
in thread 7ff0a521d940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7ff0a4c272ca]
2: (()+0x10340) [0x7ff0a333b340]
3: (gsignal()+0x39) [0x7ff0a15b6cc9]
4: (abort()+0x148) [0x7ff0a15ba0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a1ec1535]
6: (()+0x5e6d6) [0x7ff0a1ebf6d6]
7: (()+0x5d799) [0x7ff0a1ebe799]
8: (__gxx_personality_v0()+0x52a) [0x7ff0a1ebf34a]
9: (()+0xfff3) [0x7ff0a1954ff3]
10: (_Unwind_Resume()+0x57) [0x7ff0a1955517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7ff0a4d14136]
12: (()+0x2c2f80) [0x7ff0a4719f80]
13: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
15: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
16: (OSD::init()+0x26d) [0x7ff0a4799a5d]
17: (main()+0x2954) [0x7ff0a471f474]
18: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
19: (()+0x2f82b7) [0x7ff0a474f2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-03-03 14:09:01.970874 7ff0a521d940 -1 *** Caught signal (Aborted) **
in thread 7ff0a521d940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7ff0a4c272ca]
2: (()+0x10340) [0x7ff0a333b340]
3: (gsignal()+0x39) [0x7ff0a15b6cc9]
4: (abort()+0x148) [0x7ff0a15ba0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a1ec1535]
6: (()+0x5e6d6) [0x7ff0a1ebf6d6]
7: (()+0x5d799) [0x7ff0a1ebe799]
8: (__gxx_personality_v0()+0x52a) [0x7ff0a1ebf34a]
9: (()+0xfff3) [0x7ff0a1954ff3]
10: (_Unwind_Resume()+0x57) [0x7ff0a1955517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7ff0a4d14136]
12: (()+0x2c2f80) [0x7ff0a4719f80]
13: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
15: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
16: (OSD::init()+0x26d) [0x7ff0a4799a5d]
17: (main()+0x2954) [0x7ff0a471f474]
18: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
19: (()+0x2f82b7) [0x7ff0a474f2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.160375 7fd3b686c940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5719
2016-03-03 14:09:05.169748 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:05.170085 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:05.170091 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:05.170107 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:05.192222 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:05.192299 7fd3b686c940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:05.198884 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:05.200572 7fd3b686c940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:05.202443 7fd3b686c940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7fd3b686c940 time 2016-03-03 14:09:05.200585
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fd3b6362f2b]
2: (()+0x2c2f80) [0x7fd3b5d68f80]
3: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
5: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
6: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
7: (main()+0x2954) [0x7fd3b5d6e474]
8: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
9: (()+0x2f82b7) [0x7fd3b5d9e2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-24> 2016-03-03 14:09:05.158253 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perfcounters_dump hook 0x7fd3ba926050
-23> 2016-03-03 14:09:05.158272 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command 1 hook 0x7fd3ba926050
-22> 2016-03-03 14:09:05.158274 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perf dump hook 0x7fd3ba926050
-21> 2016-03-03 14:09:05.158276 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perfcounters_schema hook 0x7fd3ba926050
-20> 2016-03-03 14:09:05.158277 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command 2 hook 0x7fd3ba926050
-19> 2016-03-03 14:09:05.158279 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perf schema hook 0x7fd3ba926050
-18> 2016-03-03 14:09:05.158280 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perf reset hook 0x7fd3ba926050
-17> 2016-03-03 14:09:05.158282 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config show hook 0x7fd3ba926050
-16> 2016-03-03 14:09:05.158283 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config set hook 0x7fd3ba926050
-15> 2016-03-03 14:09:05.158285 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config get hook 0x7fd3ba926050
-14> 2016-03-03 14:09:05.158286 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config diff hook 0x7fd3ba926050
-13> 2016-03-03 14:09:05.158287 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command log flush hook 0x7fd3ba926050
-12> 2016-03-03 14:09:05.158288 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command log dump hook 0x7fd3ba926050
-11> 2016-03-03 14:09:05.158290 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command log reopen hook 0x7fd3ba926050
-10> 2016-03-03 14:09:05.160308 7fd3b686c940 0 set uid:gid to 64045:64045
-9> 2016-03-03 14:09:05.160375 7fd3b686c940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5719
-8> 2016-03-03 14:09:05.169748 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
-7> 2016-03-03 14:09:05.170085 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-6> 2016-03-03 14:09:05.170091 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-5> 2016-03-03 14:09:05.170107 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
-4> 2016-03-03 14:09:05.192222 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-3> 2016-03-03 14:09:05.192299 7fd3b686c940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
-2> 2016-03-03 14:09:05.198884 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-1> 2016-03-03 14:09:05.200572 7fd3b686c940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
0> 2016-03-03 14:09:05.202443 7fd3b686c940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7fd3b686c940 time 2016-03-03 14:09:05.200585
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fd3b6362f2b]
2: (()+0x2c2f80) [0x7fd3b5d68f80]
3: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
5: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
6: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
7: (main()+0x2954) [0x7fd3b5d6e474]
8: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
9: (()+0x2f82b7) [0x7fd3b5d9e2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.204762 7fd3b686c940 -1 *** Caught signal (Aborted) **
in thread 7fd3b686c940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7fd3b62762ca]
2: (()+0x10340) [0x7fd3b498a340]
3: (gsignal()+0x39) [0x7fd3b2c05cc9]
4: (abort()+0x148) [0x7fd3b2c090d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd3b3510535]
6: (()+0x5e6d6) [0x7fd3b350e6d6]
7: (()+0x5d799) [0x7fd3b350d799]
8: (__gxx_personality_v0()+0x52a) [0x7fd3b350e34a]
9: (()+0xfff3) [0x7fd3b2fa3ff3]
10: (_Unwind_Resume()+0x57) [0x7fd3b2fa4517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7fd3b6363136]
12: (()+0x2c2f80) [0x7fd3b5d68f80]
13: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
15: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
16: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
17: (main()+0x2954) [0x7fd3b5d6e474]
18: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
19: (()+0x2f82b7) [0x7fd3b5d9e2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-03-03 14:09:05.204762 7fd3b686c940 -1 *** Caught signal (Aborted) **
in thread 7fd3b686c940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7fd3b62762ca]
2: (()+0x10340) [0x7fd3b498a340]
3: (gsignal()+0x39) [0x7fd3b2c05cc9]
4: (abort()+0x148) [0x7fd3b2c090d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd3b3510535]
6: (()+0x5e6d6) [0x7fd3b350e6d6]
7: (()+0x5d799) [0x7fd3b350d799]
8: (__gxx_personality_v0()+0x52a) [0x7fd3b350e34a]
9: (()+0xfff3) [0x7fd3b2fa3ff3]
10: (_Unwind_Resume()+0x57) [0x7fd3b2fa4517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7fd3b6363136]
12: (()+0x2c2f80) [0x7fd3b5d68f80]
13: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
15: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
16: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
17: (main()+0x2954) [0x7fd3b5d6e474]
18: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
19: (()+0x2f82b7) [0x7fd3b5d9e2b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.392819 7f31118d0940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5807
2016-03-03 14:09:05.401988 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:05.402339 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:05.402345 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:05.402360 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:05.413273 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:05.413332 7f31118d0940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:05.421311 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:05.424378 7f31118d0940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f31118d0940 time 2016-03-03 14:09:05.422528
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f31113c6f2b]
2: (()+0x2c2f80) [0x7f3110dccf80]
3: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
5: (FileStore::mount()+0x3b42) [0x7f3111093a62]
6: (OSD::init()+0x26d) [0x7f3110e4ca5d]
7: (main()+0x2954) [0x7f3110dd2474]
8: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
9: (()+0x2f82b7) [0x7f3110e022b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-24> 2016-03-03 14:09:05.390559 7f31118d0940 5 asok(0x7f31148b8000) register_command perfcounters_dump hook 0x7f3114838050
-23> 2016-03-03 14:09:05.390587 7f31118d0940 5 asok(0x7f31148b8000) register_command 1 hook 0x7f3114838050
-22> 2016-03-03 14:09:05.390589 7f31118d0940 5 asok(0x7f31148b8000) register_command perf dump hook 0x7f3114838050
-21> 2016-03-03 14:09:05.390606 7f31118d0940 5 asok(0x7f31148b8000) register_command perfcounters_schema hook 0x7f3114838050
-20> 2016-03-03 14:09:05.390607 7f31118d0940 5 asok(0x7f31148b8000) register_command 2 hook 0x7f3114838050
-19> 2016-03-03 14:09:05.390609 7f31118d0940 5 asok(0x7f31148b8000) register_command perf schema hook 0x7f3114838050
-18> 2016-03-03 14:09:05.390610 7f31118d0940 5 asok(0x7f31148b8000) register_command perf reset hook 0x7f3114838050
-17> 2016-03-03 14:09:05.390613 7f31118d0940 5 asok(0x7f31148b8000) register_command config show hook 0x7f3114838050
-16> 2016-03-03 14:09:05.390621 7f31118d0940 5 asok(0x7f31148b8000) register_command config set hook 0x7f3114838050
-15> 2016-03-03 14:09:05.390623 7f31118d0940 5 asok(0x7f31148b8000) register_command config get hook 0x7f3114838050
-14> 2016-03-03 14:09:05.390624 7f31118d0940 5 asok(0x7f31148b8000) register_command config diff hook 0x7f3114838050
-13> 2016-03-03 14:09:05.390625 7f31118d0940 5 asok(0x7f31148b8000) register_command log flush hook 0x7f3114838050
-12> 2016-03-03 14:09:05.390632 7f31118d0940 5 asok(0x7f31148b8000) register_command log dump hook 0x7f3114838050
-11> 2016-03-03 14:09:05.390634 7f31118d0940 5 asok(0x7f31148b8000) register_command log reopen hook 0x7f3114838050
-10> 2016-03-03 14:09:05.392755 7f31118d0940 0 set uid:gid to 64045:64045
-9> 2016-03-03 14:09:05.392819 7f31118d0940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5807
-8> 2016-03-03 14:09:05.401988 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
-7> 2016-03-03 14:09:05.402339 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-6> 2016-03-03 14:09:05.402345 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-5> 2016-03-03 14:09:05.402360 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
-4> 2016-03-03 14:09:05.413273 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-3> 2016-03-03 14:09:05.413332 7f31118d0940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
-2> 2016-03-03 14:09:05.421311 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-1> 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
0> 2016-03-03 14:09:05.424378 7f31118d0940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f31118d0940 time 2016-03-03 14:09:05.422528
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f31113c6f2b]
2: (()+0x2c2f80) [0x7f3110dccf80]
3: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
5: (FileStore::mount()+0x3b42) [0x7f3111093a62]
6: (OSD::init()+0x26d) [0x7f3110e4ca5d]
7: (main()+0x2954) [0x7f3110dd2474]
8: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
9: (()+0x2f82b7) [0x7f3110e022b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.426790 7f31118d0940 -1 *** Caught signal (Aborted) **
in thread 7f31118d0940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7f31112da2ca]
2: (()+0x10340) [0x7f310f9ee340]
3: (gsignal()+0x39) [0x7f310dc69cc9]
4: (abort()+0x148) [0x7f310dc6d0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f310e574535]
6: (()+0x5e6d6) [0x7f310e5726d6]
7: (()+0x5d799) [0x7f310e571799]
8: (__gxx_personality_v0()+0x52a) [0x7f310e57234a]
9: (()+0xfff3) [0x7f310e007ff3]
10: (_Unwind_Resume()+0x57) [0x7f310e008517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7f31113c7136]
12: (()+0x2c2f80) [0x7f3110dccf80]
13: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
15: (FileStore::mount()+0x3b42) [0x7f3111093a62]
16: (OSD::init()+0x26d) [0x7f3110e4ca5d]
17: (main()+0x2954) [0x7f3110dd2474]
18: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
19: (()+0x2f82b7) [0x7f3110e022b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-03-03 14:09:05.426790 7f31118d0940 -1 *** Caught signal (Aborted) **
in thread 7f31118d0940
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7f31112da2ca]
2: (()+0x10340) [0x7f310f9ee340]
3: (gsignal()+0x39) [0x7f310dc69cc9]
4: (abort()+0x148) [0x7f310dc6d0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f310e574535]
6: (()+0x5e6d6) [0x7f310e5726d6]
7: (()+0x5d799) [0x7f310e571799]
8: (__gxx_personality_v0()+0x52a) [0x7f310e57234a]
9: (()+0xfff3) [0x7f310e007ff3]
10: (_Unwind_Resume()+0x57) [0x7f310e008517]
11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7f31113c7136]
12: (()+0x2c2f80) [0x7f3110dccf80]
13: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
15: (FileStore::mount()+0x3b42) [0x7f3111093a62]
16: (OSD::init()+0x26d) [0x7f3110e4ca5d]
17: (main()+0x2954) [0x7f3110dd2474]
18: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
19: (()+0x2f82b7) [0x7f3110e022b7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
1/ 3 keyvaluestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
