[ceph-users] Re: mgr services frequently crash on nodes 2,3,4

[陶冬冬]

This mgr assert failure is fixed at https://github.com/ceph/ceph/pull/46688
You can upgrade to 16.2.13 to get the fix.


Eugen Block  于2023年8月3日周四 14:57写道：

> Can you query those config options yourself?
>
> storage01:~ # ceph config get mgr mgr/dashboard/standby_behaviour
> storage01:~ # ceph config get mgr mgr/dashboard/AUDIT_API_ENABLED
>
> I'm not sure if those are responsible for the crash though.
>
> Zitat von "Adiga, Anantha" :
>
> > Hi,
> >
> > Mgr service crash frequently on nodes 2 3 and 4  with the same
> > condition after the 4th node was added.
> >
> > root@zp3110b001a0104:/# ceph crash stat
> > 19 crashes recorded
> > 16 older than 1 days old:
> > 2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189
> > 2023-07-29T03:35:32.055174Z_a2ee1e23-5f41-4dbe-86ff-643fbf870dc9
> > 2023-07-29T14:34:13.752432Z_39b6a0d9-1bc3-4481-9a14-c92fea6c2710
> > 2023-07-30T03:02:57.510867Z_df595e04-0ac2-4e3d-93be-a7225348ea19
> > 2023-07-30T06:20:09.322530Z_0c2485f8-281c-4440-8b08-89b08a669de4
> > 2023-07-30T10:16:46.798405Z_79082f37-ee08-4a2b-84d1-d96c4026f321
> > 2023-07-30T10:16:46.843441Z_788391d6-3278-48c4-a95b-1934ee3265c1
> > 2023-07-31T02:26:55.903966Z_416a1e94-a8e1-4057-a683-a907faf400a1
> > 2023-07-31T04:40:10.216044Z_bef9d811-4e92-45cd-bcd7-3282962c8dfe
> > 2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6
> > 2023-07-31T09:22:12.527968Z_f136c93b-7156-4176-a734-66a5a62513a4
> > 2023-07-31T15:22:08.417988Z_b80c6255-5eb3-41dd-b0b1-8bc5b070094f
> > 2023-07-31T23:05:16.589501Z_20ed8ef9-a478-49de-a371-08ea7a9937e5
> > 2023-08-01T01:26:01.911387Z_670f9e3c-7fbe-497f-9f0b-abeaefd8f2b3
> > 2023-08-01T01:51:39.759874Z_ff8206e4-34aa-44fe-82ac-7339e6714bb7
> > 2023-08-01T01:56:21.955706Z_98c86cdd-45ec-47dc-8f0c-2e5e09731db8
> > 7 older than 3 days old:
> > 2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189
> > 2023-07-29T03:35:32.055174Z_a2ee1e23-5f41-4dbe-86ff-643fbf870dc9
> > 2023-07-29T14:34:13.752432Z_39b6a0d9-1bc3-4481-9a14-c92fea6c2710
> > 2023-07-30T03:02:57.510867Z_df595e04-0ac2-4e3d-93be-a7225348ea19
> > 2023-07-30T06:20:09.322530Z_0c2485f8-281c-4440-8b08-89b08a669de4
> > 2023-07-30T10:16:46.798405Z_79082f37-ee08-4a2b-84d1-d96c4026f321
> > 2023-07-30T10:16:46.843441Z_788391d6-3278-48c4-a95b-1934ee3265c1
> >
> > root@zp3110b001a0104
> :/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6# root@zp3110b001a0104:/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6#>
> cat
> > meta
> > {
> > "crash_id":
> > "2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6",
> > "timestamp": "2023-07-31T08:44:20.893344Z",
> > "process_name": "ceph-mgr",
> > "entity_name": "mgr.zp3110b001a0104.tmbkzq",
> > "ceph_version": "16.2.5",
> > "utsname_hostname": "zp3110b001a0104",
> > "utsname_sysname": "Linux",
> > "utsname_release": "5.4.0-153-generic",
> > "utsname_version": "#170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023",
> > "utsname_machine": "x86_64",
> > "os_name": "CentOS Linux",
> > "os_id": "centos",
> > "os_version_id": "8",
> > "os_version": "8",
> > "assert_condition": "pending_service_map.epoch > service_map.epoch",
> > "assert_func": "DaemonServer::got_service_map():: > ServiceMap&)>",
> > "assert_file":
> >
> "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc",
> > "assert_line": 2932,
> > "assert_thread_name": "ms_dispatch",
> > "assert_msg":
> >
> "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc:
> In function 'DaemonServer::got_service_map()::'
> thread 7f127440a700 time
> 2023-07-31T08:44:20.887150+\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc:
> 2932: FAILED ceph_assert(pending_service_map.epoch >
> > service_map.epoch)\n",
> > "backtrace": [
> > "/lib64/libpthread.so.0(+0x12b20) [0x7f127c611b20]",
> > "gsignal()",
> > "abort()",
> > "(ceph::__ceph_assert_fail(char const*, char const*, int,
> > char const*)+0x1a9) [0x7f127da26b75]",
> > "/usr/lib64/ceph/libceph-common.so.2(+0x276d3e)
> [0x7f127da26d3e]",
> > "(DaemonServer::got_service_map()+0xb2d) [0x5625aee23a4d]",
> >
> > "(Mgr::handle_service_map(boost::intrusive_ptr)+0x1b6)
> > [0x5625aee527c6]",
> > "(Mgr::ms_dispatch2(boost::intrusive_ptr
> > const&)+0x894) [0x5625aee55424]",
> > "(MgrStandby::ms_dispatch2(boost::intrusive_ptr
> > const&)+0xb0) 

[ceph-users] Re: mgr services frequently crash on nodes 2,3,4

[Eugen Block]

Can you query those config options yourself?

storage01:~ # ceph config get mgr mgr/dashboard/standby_behaviour
storage01:~ # ceph config get mgr mgr/dashboard/AUDIT_API_ENABLED

I'm not sure if those are responsible for the crash though.

Zitat von "Adiga, Anantha" :


Hi,

Mgr service crash frequently on nodes 2 3 and 4  with the same  
condition after the 4th node was added.


root@zp3110b001a0104:/# ceph crash stat
19 crashes recorded
16 older than 1 days old:
2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189
2023-07-29T03:35:32.055174Z_a2ee1e23-5f41-4dbe-86ff-643fbf870dc9
2023-07-29T14:34:13.752432Z_39b6a0d9-1bc3-4481-9a14-c92fea6c2710
2023-07-30T03:02:57.510867Z_df595e04-0ac2-4e3d-93be-a7225348ea19
2023-07-30T06:20:09.322530Z_0c2485f8-281c-4440-8b08-89b08a669de4
2023-07-30T10:16:46.798405Z_79082f37-ee08-4a2b-84d1-d96c4026f321
2023-07-30T10:16:46.843441Z_788391d6-3278-48c4-a95b-1934ee3265c1
2023-07-31T02:26:55.903966Z_416a1e94-a8e1-4057-a683-a907faf400a1
2023-07-31T04:40:10.216044Z_bef9d811-4e92-45cd-bcd7-3282962c8dfe
2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6
2023-07-31T09:22:12.527968Z_f136c93b-7156-4176-a734-66a5a62513a4
2023-07-31T15:22:08.417988Z_b80c6255-5eb3-41dd-b0b1-8bc5b070094f
2023-07-31T23:05:16.589501Z_20ed8ef9-a478-49de-a371-08ea7a9937e5
2023-08-01T01:26:01.911387Z_670f9e3c-7fbe-497f-9f0b-abeaefd8f2b3
2023-08-01T01:51:39.759874Z_ff8206e4-34aa-44fe-82ac-7339e6714bb7
2023-08-01T01:56:21.955706Z_98c86cdd-45ec-47dc-8f0c-2e5e09731db8
7 older than 3 days old:
2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189
2023-07-29T03:35:32.055174Z_a2ee1e23-5f41-4dbe-86ff-643fbf870dc9
2023-07-29T14:34:13.752432Z_39b6a0d9-1bc3-4481-9a14-c92fea6c2710
2023-07-30T03:02:57.510867Z_df595e04-0ac2-4e3d-93be-a7225348ea19
2023-07-30T06:20:09.322530Z_0c2485f8-281c-4440-8b08-89b08a669de4
2023-07-30T10:16:46.798405Z_79082f37-ee08-4a2b-84d1-d96c4026f321
2023-07-30T10:16:46.843441Z_788391d6-3278-48c4-a95b-1934ee3265c1

root@zp3110b001a0104:/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6# cat  
meta

{
"crash_id":  
"2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6",

"timestamp": "2023-07-31T08:44:20.893344Z",
"process_name": "ceph-mgr",
"entity_name": "mgr.zp3110b001a0104.tmbkzq",
"ceph_version": "16.2.5",
"utsname_hostname": "zp3110b001a0104",
"utsname_sysname": "Linux",
"utsname_release": "5.4.0-153-generic",
"utsname_version": "#170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023",
"utsname_machine": "x86_64",
"os_name": "CentOS Linux",
"os_id": "centos",
"os_version_id": "8",
"os_version": "8",
"assert_condition": "pending_service_map.epoch > service_map.epoch",
"assert_func": "DaemonServer::got_service_map()::ServiceMap&)>",
"assert_file":  
"/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc",

"assert_line": 2932,
"assert_thread_name": "ms_dispatch",
"assert_msg":  
"/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc: In function 'DaemonServer::got_service_map()::' thread 7f127440a700 time 2023-07-31T08:44:20.887150+\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc: 2932: FAILED ceph_assert(pending_service_map.epoch >  
service_map.epoch)\n",

"backtrace": [
"/lib64/libpthread.so.0(+0x12b20) [0x7f127c611b20]",
"gsignal()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int,  
char const*)+0x1a9) [0x7f127da26b75]",

"/usr/lib64/ceph/libceph-common.so.2(+0x276d3e) [0x7f127da26d3e]",
"(DaemonServer::got_service_map()+0xb2d) [0x5625aee23a4d]",
 
"(Mgr::handle_service_map(boost::intrusive_ptr)+0x1b6)  
[0x5625aee527c6]",
"(Mgr::ms_dispatch2(boost::intrusive_ptr  
const&)+0x894) [0x5625aee55424]",
"(MgrStandby::ms_dispatch2(boost::intrusive_ptr  
const&)+0xb0) [0x5625aee5ec10]",

"(DispatchQueue::entry()+0x126a) [0x7f127dc610ca]",
"(DispatchQueue::DispatchThread::entry()+0x11) [0x7f127dd11591]",
"/lib64/libpthread.so.0(+0x814a) [0x7f127c60714a]",
"clone()"
]
}