Hi all,
we use ceph(hammer) + openstack(mitaka) in my datacenter and there are 300
osds and 3. Because the accident datacenter is powered off, all the servers are
shut down. when power returns to normal ,we start 3 mon service at first, About
two hours later we start 500 osd service,and later cluster is ok.
but one day later, there are "auth: could not find secret_id=6403" error on
several osd hosts,and there are op is blocked
2019-05-27 19:23:44.316416 7fb75451e700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.210:6816/1004678 pipe(0x128f5000 sd=586 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4f8c0).accept connect_seq 5 vs existing 5 state standby
2019-05-27 19:23:44.316519 7fb755029700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.203:6834/1004655 pipe(0x173dd000 sd=584 :6832 s=0 pgs=0 cs=0 l=0
c=0x27472ec0).accept connect_seq 8 vs existing 7 state standby
2019-05-27 19:23:44.316561 7fb788d5c700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.211:6808/5206 pipe(0x144db000 sd=587 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f51700).accept connect_seq 7 vs existing 7 state standby
2019-05-27 19:23:44.316656 7fb77c198700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.219:6818/8946 pipe(0x1f575000 sd=588 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4e580).accept connect_seq 7 vs existing 7 state standby
2019-05-27 19:23:44.316719 7fb78aa79700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.213:6810/5387 pipe(0x1f570000 sd=93 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f512e0).accept connect_seq 5 vs existing 5 state standby
2019-05-27 19:23:44.316852 7fb754922700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.203:6828/1004835 pipe(0x128f0000 sd=585 :6832 s=0 pgs=0 cs=0 l=0
c=0x21d65b20).accept connect_seq 4 vs existing 3 state standby
2019-05-27 19:23:44.316929 7fb788d5c700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.211:6808/5206 pipe(0x144db000 sd=587 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f51700).accept connect_seq 8 vs existing 7 state standby
2019-05-27 19:23:44.317004 7fb75451e700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.210:6816/1004678 pipe(0x128f5000 sd=586 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4f8c0).accept connect_seq 6 vs existing 5 state standby
2019-05-27 19:23:44.317148 7fb78aa79700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.213:6810/5387 pipe(0x1f570000 sd=93 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f512e0).accept connect_seq 6 vs existing 5 state standby
2019-05-27 19:23:44.317207 7fb77c198700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.219:6818/8946 pipe(0x1f575000 sd=588 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4e580).accept connect_seq 8 vs existing 7 state standby
2019-05-27 19:28:51.828430 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:51.828446 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:51.828453 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f50aa0).accept: got bad authorizer
2019-05-27 19:28:51.829282 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:51.829296 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:51.829303 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x21ba7000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4f4a0).accept: got bad authorizer
2019-05-27 19:28:52.030139 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:52.030153 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:52.030161 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x20be9000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f51860).accept: got bad authorizer
2019-05-27 19:28:52.431002 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:52.431017 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:52.431024 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4f600).accept: got bad authorizer
2019-05-27 19:28:53.231883 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:53.231896 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:53.231903 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x21ba7000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27f4e9a0).accept: got bad authorizer
2019-05-27 19:28:54.832790 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:54.832805 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:54.832812 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x16f579c0).accept: got bad authorizer
2019-05-27 19:28:58.033720 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:28:58.033734 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:28:58.033740 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x20be9000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27471ce0).accept: got bad authorizer
2019-05-27 19:29:04.434805 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:29:04.434819 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:29:04.434826 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x21ba7000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0xee4d440).accept: got bad authorizer
2019-05-27 19:29:17.235957 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:29:17.235971 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:29:17.235980 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x21d64aa0).accept: got bad authorizer
2019-05-27 19:29:32.237179 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:29:32.237193 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:29:32.237200 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x20be9000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x27211b20).accept: got bad authorizer
2019-05-27 19:29:47.238267 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:29:47.238282 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:29:47.238289 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x21ba7000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x13f91c80).accept: got bad authorizer
2019-05-27 19:30:02.239268 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:30:02.239281 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:30:02.239288 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x13f8edc0).accept: got bad authorizer
2019-05-27 19:30:17.240508 7fb756e47700 0 auth: could not find secret_id=6403
2019-05-27 19:30:17.240525 7fb756e47700 0 cephx: verify_authorizer could not
get service secret for service osd secret_id=6403
2019-05-27 19:30:17.240535 7fb756e47700 0 -- 10.22.9.197:6832/1005404 >>
10.22.9.216:6816/28337 pipe(0x20be9000 sd=145 :6832 s=0 pgs=0 cs=0 l=0
c=0x197a8f20).accept: got bad authorizer
2019-05-27 19:30:32.241585 7fb756e47700 0 auth: could not find secret_id=6403
Why does this happen? and how to do.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
