I have not configured anything for the msd except this
[mds]
# 100k+ files in 2 folders
mds bal fragment size max = 120000
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
mds_session_blacklist_on_timeout = false
mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 8000000000
# faster fail over?
#mds beacon grace = 5
-----Original Message-----
From: Eugen Block [mailto:ebl...@nde.ag]
Sent: dinsdag 21 mei 2019 10:18
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Is a not active mds doing something?
Hi Marc,
have you configured the other MDS to be standby-replay for the active
MDS? I have three MDS servers, one is active, the second is
active-standby and the third just standby. If the active fails, the
second takes over within seconds. This is what I have in my ceph.conf:
[mds.<YOUR MDS>]
mds_standby_replay = true
mds_standby_for_rank = 0
Regards,
Eugen
Zitat von Marc Roos <m.r...@f1-outsourcing.eu>:
> Should a not active mds be doing something??? When I restarted the not
> active mds.c, My client io on the fs_data pool disappeared.
>
>
> services:
> mon: 3 daemons, quorum a,b,c
> mgr: c(active), standbys: a, b
> mds: cephfs-1/1/1 up {0=a=up:active}, 1 up:standby
> osd: 32 osds: 32 up, 32 in
> rgw: 2 daemons active
>
>
>
> -----Original Message-----
> From: Marc Roos
> Sent: dinsdag 21 mei 2019 10:01
> To: ceph-users@lists.ceph.com; Marc Roos
> Subject: RE: [ceph-users] cephfs causing high load on vm, taking down
> 15 min later another cephfs vm
>
>
> I have evicted all client connections and have still high load on
> osd's
>
> And ceph osd pool stats shows still client activity?????
>
> pool fs_data id 20
> client io 565KiB/s rd, 120op/s rd, 0op/s wr
>
>
>
>
> -----Original Message-----
> From: Marc Roos
> Sent: dinsdag 21 mei 2019 9:51
> To: ceph-users@lists.ceph.com; Marc Roos
> Subject: RE: [ceph-users] cephfs causing high load on vm, taking down
> 15 min later another cephfs vm
>
>
> I have got this today again????? I cannot unmount the filesystem and
> looks like some osd's are having 100% cpu utilization?
>
>
> -----Original Message-----
> From: Marc Roos
> Sent: maandag 20 mei 2019 12:42
> To: ceph-users
> Subject: [ceph-users] cephfs causing high load on vm, taking down 15
> min later another cephfs vm
>
>
>
> I got my first problem with cephfs in a production environment. Is it
> possible from these logfiles to deduct what happened?
>
> svr1 is connected to ceph client network via switch
> svr2 vm is collocated on c01 node.
> c01 has osd's and the mon.a colocated.
>
> svr1 was the first to report errors at 03:38:44. I have no error
> messages reported of a network connection problem by any of the ceph
> nodes. I have nothing in dmesg on c01.
>
> [@c01 ~]# cat /etc/redhat-release
> CentOS Linux release 7.6.1810 (Core)
> [@c01 ~]# uname -a
> Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC
> 2019
>
> x86_64 x86_64 x86_64 GNU/Linux
> [@c01 ~]# ceph versions
> {
> "mon": {
> "ceph version 12.2.12
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 3
> },
> "mgr": {
> "ceph version 12.2.12
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 3
> },
> "osd": {
> "ceph version 12.2.12
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 32
> },
> "mds": {
> "ceph version 12.2.12
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 2
> },
> "rgw": {
> "ceph version 12.2.12
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 2
> },
> "overall": {
> "ceph version 12.2.12
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 42
> }
> }
>
>
>
>
> [0] svr1 messages
> May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
> May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
> May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
> May 20 03:38:45 svr1 kernel: last message repeated 5 times May 20
> 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error May 20
> 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session lost,
> hunting for new mon May 20 03:38:45 svr1 kernel: last message repeated
> 5 times May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789
> io error May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789
> session lost, hunting for new mon May 20 03:38:45 svr1 kernel:
> libceph: mon1 192.168.x.112:6789 session established May 20 03:38:45
> svr1 kernel: libceph: mon1 192.168.x.112:6789 session established May
> 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May
> 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May
> 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error May
> 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session
> lost, hunting for new mon May 20 03:38:45 svr1 kernel: libceph: mon1
> 192.168.x.112:6789 io error May 20 03:38:45 svr1 kernel: libceph: mon1
> 192.168.x.112:6789 session lost, hunting for new mon May 20 03:38:45
> svr1 kernel: libceph: mon0 192.168.x.111:6789 session established May
> 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session
> established May 20 03:38:45 svr1 kernel: libceph: mon0
> 192.168.x.111:6789 io error May 20 03:38:45 svr1 kernel: libceph: mon0
> 192.168.x.111:6789 session lost, hunting for new mon May 20 03:38:45
> svr1 kernel: libceph: mon0 192.168.x.111:6789 io error May 20 03:38:45
> svr1 kernel: libceph: mon0 192.168.x.111:6789 session lost, hunting
> for new mon May 20 03:38:45 svr1 kernel: libceph: mon2
> 192.168.x.113:6789 session established May 20 03:38:45 svr1 kernel:
> libceph: mon2 192.168.x.113:6789 session established May 20 03:38:45
> svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May 20 03:38:45
> svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May 20 03:38:45
> svr1 kernel: libceph: mon2 192.168.x.113:6789 io error May 20 03:38:45
> svr1 kernel: libceph: mon2 192.168.x.113:6789 session lost, hunting
> for new mon May 20 03:38:45 svr1 kernel: libceph: mon2
> 192.168.x.113:6789 io error May 20 03:38:45 svr1 kernel: libceph: mon2
> 192.168.x.113:6789 session lost, hunting for new mon May 20 03:38:45
> svr1 kernel: libceph: mon0 192.168.x.111:6789 session established May
> 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error May
> 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session
> lost, hunting for new mon May 20 03:38:45 svr1 kernel: libceph: mon0
> 192.168.x.111:6789 session established
>
>
> [1] svr2 messages
> May 20 03:40:01 svr2 systemd: Stopping User Slice of root.
> May 20 03:40:01 svr2 systemd: Removed slice User Slice of root.
> May 20 03:40:01 svr2 systemd: Stopping User Slice of root.
> May 20 03:50:01 svr2 systemd: Created slice User Slice of root.
> May 20 03:50:01 svr2 systemd: Created slice User Slice of root.
> May 20 03:50:01 svr2 systemd: Starting User Slice of root.
> May 20 03:50:01 svr2 systemd: Starting User Slice of root.
> May 20 03:50:01 svr2 systemd: Started Session 9442 of user root.
> May 20 03:50:01 svr2 systemd: Started Session 9442 of user root.
> May 20 03:50:01 svr2 systemd: Starting Session 9442 of user root.
> May 20 03:50:01 svr2 systemd: Starting Session 9442 of user root.
> May 20 03:50:01 svr2 systemd: Removed slice User Slice of root.
> May 20 03:50:01 svr2 systemd: Removed slice User Slice of root.
> May 20 03:50:01 svr2 systemd: Stopping User Slice of root.
> May 20 03:50:01 svr2 systemd: Stopping User Slice of root.
> May 20 03:53:37 svr2 kernel: libceph: osd9 192.168.x.112:6812 io error
> May 20 03:53:37 svr2 kernel: libceph: osd9 192.168.x.112:6812 io error
> May 20 03:53:37 svr2 kernel: last message repeated 3 times May 20
> 03:53:37 svr2 kernel: libceph: mon1 192.168.x.112:6789 io error May 20
> 03:53:37 svr2 kernel: last message repeated 3 times May 20 03:53:37
> svr2 kernel: libceph: mon1 192.168.x.112:6789 io error May 20 03:53:37
> svr2 kernel: libceph: mon1 192.168.x.112:6789 session lost, hunting
> for new mon May 20 03:53:37 svr2 kernel: libceph: mon1
> 192.168.x.112:6789 session lost, hunting for new mon May 20 03:53:37
> svr2 kernel: libceph: mon2 192.168.x.113:6789 session established May
> 20 03:53:37 svr2 kernel: libceph: mon2 192.168.x.113:6789 session
> established May 20 03:53:37 svr2 kernel: libceph: mon2
> 192.168.x.113:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon2
> 192.168.x.113:6789 session lost, hunting for new mon May 20 03:53:37
> svr2 kernel: libceph: mon2 192.168.x.113:6789 io error May 20 03:53:37
> svr2 kernel: libceph: mon2 192.168.x.113:6789 session lost, hunting
> for new mon May 20 03:53:37 svr2 kernel: libceph: osd9
> 192.168.x.112:6812 io error May 20 03:53:37 svr2 kernel: libceph: osd9
> 192.168.x.112:6812 io error May 20 03:53:37 svr2 kernel: libceph: mon1
> 192.168.x.112:6789 session established May 20 03:53:37 svr2 kernel:
> libceph: mon1 192.168.x.112:6789 session established May 20 03:53:37
> svr2 kernel: libceph: mon1 192.168.x.112:6789 io error May 20 03:53:37
> svr2 kernel: libceph: mon1 192.168.x.112:6789 session lost, hunting
> for new mon May 20 03:53:37 svr2 kernel: libceph: mon1
> 192.168.x.112:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon1
> 192.168.x.112:6789 session lost, hunting for new mon May 20 03:53:37
> svr2 kernel: libceph: mon0 192.168.x.111:6789 session established May
> 20 03:53:37 svr2 kernel: libceph: mon0 192.168.x.111:6789 session
> established May 20 03:53:37 svr2 kernel: libceph: mon0
> 192.168.x.111:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon0
> 192.168.x.111:6789 session lost, hunting for new mon May 20 03:53:37
> svr2 kernel: libceph: mon0 192.168.x.111:6789 io error May 20 03:53:37
> svr2 kernel: libceph: mon0 192.168.x.111:6789 session lost, hunting
> for new mon May 20 03:53:37 svr2 kernel: libceph: mon2
> 192.168.x.113:6789 session established May 20 03:53:37 svr2 kernel:
> libceph: mon2 192.168.x.113:6789 session established May 20 03:53:37
> svr2 kernel: libceph: mon2 192.168.x.113:6789 io error May 20 03:53:37
> svr2 kernel: libceph: mon2 192.168.x.113:6789 session lost, hunting
> for new mon May 20 03:53:37 svr2 kernel: libceph: mon2
> 192.168.x.113:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon2
> 192.168.x.113:6789 session lost, hunting for new mon
>
> [2] osd.0 log
> 2019-05-20 03:38:46.358270 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55780c19e000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:38:56.155141 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55776afb6000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:38:56.476312 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557797300800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:39:35.050674 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55784c099000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:41:46.605523 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55778dba8000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:42:05.201417 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.112:0/2834749548 conn(0x5578179cf800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:42:18.275703 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557773ccf800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:42:18.493838 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557898a90000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:42:18.728962 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55776afba800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:42:19.242145 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557762c8f000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:43:41.492125 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x5577aa28d800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:45:40.006405 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557778d1d800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:45:40.736819 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x5577a4224800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:47:08.368138 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557778d1d800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:48:24.848331 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x5577f0819800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:50:44.442386 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55777ef3c000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:51:11.352119 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55779e445000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:51:18.615690 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55779e445000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:51:57.887069 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55777ef3c000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:51:58.109173 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557769206000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:51:58.364811 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x5577aa28d800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:51:59.323286 7f9208e6a700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x557773cce000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:57:22.060831 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55786cf37800 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 03:57:48.793125 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.114:0/4006640003 conn(0x5577972ff000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 04:07:07.135252 7f9207e68700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55779b319000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1) 2019-05-20 04:07:07.394734 7f9208669700 0 --
> 192.168.x.111:6814/3478915
>
>>> 192.168.x.43:0/1827964483 conn(0x55779529e000 :6814
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=1).handle_connect_msg accept replacing existing (lossy) channel (new
> one lossy=1)
>
>
> [3] mon.a log
> 2019-05-20 03:36:12.418662 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316172418658, "job": 57455, "event":
> "table_file_deletion", "file_number": 1788761} 2019-05-20
> 03:36:12.431694 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316172431690, "job": 57455, "event":
> "table_file_deletion", "file_number": 1788760} 2019-05-20
> 03:36:12.444811 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316172444808, "job": 57455, "event":
> "table_file_deletion", "file_number": 1788759} 2019-05-20
> 03:36:12.458658 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316172458654, "job": 57455, "event":
> "table_file_deletion", "file_number": 1788758} 2019-05-20
> 03:36:12.472801 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316172472797, "job": 57455, "event":
> "table_file_deletion", "file_number": 1788757} 2019-05-20
> 03:36:12.487007 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316172486995, "job": 57455, "event":
> "table_file_deletion", "file_number": 1788756} 2019-05-20
> 03:36:12.487096 7f1b08e2b700 4 rocksdb: (Original Log Time
> 2019/05/20-03:36:12.487089)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu
> sh.cc:1456] Compaction nothing to do
> 2019-05-20 03:36:15.272950 7f1b0a62e700 1 mon.a@0(leader).osd e64430
> e64430: 32 total, 32 up, 32 in
> 2019-05-20 03:36:15.298861 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64430: 32 total, 32 up, 32 in 2019-05-20
> 03:36:17.013839 7f1b1163c700 0
> mon.a@0(leader).data_health(7108) update_stats avail 91% total
> 50.0GiB, used 4.05GiB, avail 45.9GiB 2019-05-20 03:36:19.738495
> 7f1b0a62e700 1 mon.a@0(leader).osd e64431
> e64431: 32 total, 32 up, 32 in
> 2019-05-20 03:36:19.765029 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64431: 32 total, 32 up, 32 in 2019-05-20
> 03:36:25.302195 7f1b0a62e700 1 mon.a@0(leader).osd e64432
> e64432: 32 total, 32 up, 32 in
> 2019-05-20 03:36:25.328639 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64432: 32 total, 32 up, 32 in 2019-05-20
> 03:36:29.741140 7f1b0a62e700 1 mon.a@0(leader).osd e64433
> e64433: 32 total, 32 up, 32 in
> 2019-05-20 03:36:29.768034 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64433: 32 total, 32 up, 32 in 2019-05-20
> 03:36:35.343530 7f1b0a62e700 1 mon.a@0(leader).osd e64434
> e64434: 32 total, 32 up, 32 in
> 2019-05-20 03:36:35.370266 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64434: 32 total, 32 up, 32 in 2019-05-20
> 03:36:39.738502 7f1b0a62e700 1 mon.a@0(leader).osd e64435
> e64435: 32 total, 32 up, 32 in
> 2019-05-20 03:36:39.765465 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64435: 32 total, 32 up, 32 in 2019-05-20
> 03:36:45.371359 7f1b0a62e700 1 mon.a@0(leader).osd e64436
> e64436: 32 total, 32 up, 32 in
> 2019-05-20 03:36:45.398034 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64436: 32 total, 32 up, 32 in 2019-05-20
> 03:36:49.738038 7f1b0a62e700 1 mon.a@0(leader).osd e64437
> e64437: 32 total, 32 up, 32 in
> 2019-05-20 03:36:49.765428 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64437: 32 total, 32 up, 32 in 2019-05-20
> 03:36:55.418010 7f1b0a62e700 1 mon.a@0(leader).osd e64438
> e64438: 32 total, 32 up, 32 in
> 2019-05-20 03:36:55.446785 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64438: 32 total, 32 up, 32 in 2019-05-20
> 03:36:59.738000 7f1b0a62e700 1 mon.a@0(leader).osd e64439
> e64439: 32 total, 32 up, 32 in
> 2019-05-20 03:36:59.765632 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64439: 32 total, 32 up, 32 in 2019-05-20
> 03:37:05.423528 7f1b0a62e700 1 mon.a@0(leader).osd e64440
> e64440: 32 total, 32 up, 32 in
> 2019-05-20 03:37:05.449968 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64440: 32 total, 32 up, 32 in 2019-05-20
> 03:37:06.513317 7f1b0a62e700 1 mon.a@0(leader).osd e64441
> e64441: 32 total, 32 up, 32 in
> 2019-05-20 03:37:06.539643 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64441: 32 total, 32 up, 32 in 2019-05-20
> 03:37:09.851351 7f1b0a62e700 1 mon.a@0(leader).osd e64442
> e64442: 32 total, 32 up, 32 in
> 2019-05-20 03:37:09.877349 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64442: 32 total, 32 up, 32 in 2019-05-20
> 03:37:15.446739 7f1b0a62e700 1 mon.a@0(leader).osd e64443
> e64443: 32 total, 32 up, 32 in
> 2019-05-20 03:37:15.447538 7f1b0a62e700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_write.cc:725
> ] [default] New memtable created with log file: #1788785. Immutable
> memtables: 0.
> 2019-05-20 03:37:15.470599 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:37:15.447607)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu sh.cc:1158] Calling FlushMemTableToOutputFile with column family
> [default], flush slots available 1, compaction slots allowed 1,
> compaction slots scheduled 1 2019-05-20 03:37:15.470613 7f1b0962c700
> 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:264]
> [default] [JOB 57457] Flushing memtable with next logfile: 1788785
> 2019-05-20 03:37:15.470629 7f1b0962c700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316235470621, "job": 57457, "event":
> "flush_started", "num_memtables": 1, "num_entries": 550,
"num_deletes":
> 0, "memory_usage": 32932624}
> 2019-05-20 03:37:15.470632 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:293]
> [default] [JOB 57457] Level-0 flush table #1788786: started 2019-05-20
> 03:37:15.474323 7f1b0a62e700 0 log_channel(cluster) log [DBG] :
> osdmap e64443: 32 total, 32 up, 32 in 2019-05-20 03:37:15.622750
> 7f1b0962c700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316235622730, "cf_name": "default", "job": 57457,
> "event": "table_file_creation", "file_number": 1788786, "file_size":
> 29021225, "table_properties": {"data_size": 28956685, "index_size":
> 5597, "filter_size": 58018, "raw_key_size": 4239,
> "raw_average_key_size": 22, "raw_value_size": 28949846,
> "raw_average_value_size": 155644, "num_data_blocks": 158,
"num_entries":
>
> 186, "filter_policy_name": "rocksdb.BuiltinBloomFilter",
"kDeletedKeys":
>
> "0", "kMergeOperands": "0"}}
> 2019-05-20 03:37:15.622780 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:319]
> [default] [JOB 57457] Level-0 flush table #1788786: 29021225 bytes OK
> 2019-05-20 03:37:15.645271 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:37:15.622795)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:360
> ] [default] Level-0 commit table #1788786 started 2019-05-20
> 03:37:15.645286 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:37:15.645176)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:383
> ] [default] Level-0 commit table #1788786: memtable #1 done 2019-05-20
> 03:37:15.645297 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:37:15.645212) EVENT_LOG_v1 {"time_micros":
> 1558316235645199, "job": 57457, "event": "flush_finished",
"lsm_state":
> [1, 0, 0, 0, 0, 0, 19], "immutable_memtables": 0} 2019-05-20
> 03:37:15.645302 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:37:15.645244)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu sh.cc:132] [default] Level summary: base level 5 max bytes base
> 69194285
>
> files[1 0 0 0 0 0 19] max score 0.25
> 2019-05-20 03:37:15.645318 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388
> ] [JOB 57457] Try to delete WAL files size 28975051, prev total WAL
> file size 28975092, number of live WAL files 2.
> 2019-05-20 03:37:16.596003 7f1b0a62e700 1 mon.a@0(leader).osd e64444
> e64444: 32 total, 32 up, 32 in
> 2019-05-20 03:37:16.621741 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64444: 32 total, 32 up, 32 in 2019-05-20
> 03:37:17.014200 7f1b1163c700 0
> mon.a@0(leader).data_health(7108) update_stats avail 91% total
> 50.0GiB, used 4.08GiB, avail 45.9GiB 2019-05-20 03:37:19.822194
> 7f1b0a62e700 1 mon.a@0(leader).osd e64445
> e64445: 32 total, 32 up, 32 in
> 2019-05-20 03:37:19.848514 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64445: 32 total, 32 up, 32 in 2019-05-20
> 03:37:25.473280 7f1b0a62e700 1 mon.a@0(leader).osd e64446
> e64446: 32 total, 32 up, 32 in
> 2019-05-20 03:37:25.499555 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64446: 32 total, 32 up, 32 in 2019-05-20
> 03:37:26.630462 7f1b0a62e700 1 mon.a@0(leader).osd e64447
> e64447: 32 total, 32 up, 32 in
> 2019-05-20 03:37:26.656497 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64447: 32 total, 32 up, 32 in 2019-05-20
> 03:37:29.856077 7f1b0a62e700 1 mon.a@0(leader).osd e64448
> e64448: 32 total, 32 up, 32 in
> 2019-05-20 03:37:29.882033 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64448: 32 total, 32 up, 32 in 2019-05-20
> 03:37:35.503666 7f1b0a62e700 1 mon.a@0(leader).osd e64449
> e64449: 32 total, 32 up, 32 in
> 2019-05-20 03:37:35.535606 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64449: 32 total, 32 up, 32 in 2019-05-20
> 03:37:36.662389 7f1b0a62e700 1 mon.a@0(leader).osd e64450
> e64450: 32 total, 32 up, 32 in
> 2019-05-20 03:37:36.688901 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64450: 32 total, 32 up, 32 in 2019-05-20
> 03:37:39.889782 7f1b0a62e700 1 mon.a@0(leader).osd e64451
> e64451: 32 total, 32 up, 32 in
> 2019-05-20 03:37:39.916350 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64451: 32 total, 32 up, 32 in 2019-05-20
> 03:37:45.534370 7f1b0a62e700 1 mon.a@0(leader).osd e64452
> e64452: 32 total, 32 up, 32 in
> 2019-05-20 03:37:45.561000 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64452: 32 total, 32 up, 32 in 2019-05-20
> 03:37:49.755239 7f1b0a62e700 1 mon.a@0(leader).osd e64453
> e64453: 32 total, 32 up, 32 in
> 2019-05-20 03:37:49.782078 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64453: 32 total, 32 up, 32 in 2019-05-20
> 03:37:50.904926 7f1b0a62e700 1 mon.a@0(leader).osd e64454
> e64454: 32 total, 32 up, 32 in
> 2019-05-20 03:37:50.930940 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64454: 32 total, 32 up, 32 in 2019-05-20
> 03:37:55.567119 7f1b0a62e700 1 mon.a@0(leader).osd e64455
> e64455: 32 total, 32 up, 32 in
> 2019-05-20 03:37:55.594617 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64455: 32 total, 32 up, 32 in 2019-05-20
> 03:37:59.736903 7f1b0a62e700 1 mon.a@0(leader).osd e64456
> e64456: 32 total, 32 up, 32 in
> 2019-05-20 03:37:59.763752 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64456: 32 total, 32 up, 32 in 2019-05-20
> 03:38:05.584960 7f1b0a62e700 1 mon.a@0(leader).osd e64457
> e64457: 32 total, 32 up, 32 in
> 2019-05-20 03:38:05.616558 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64457: 32 total, 32 up, 32 in 2019-05-20
> 03:38:09.736670 7f1b0a62e700 1 mon.a@0(leader).osd e64458
> e64458: 32 total, 32 up, 32 in
> 2019-05-20 03:38:09.762625 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64458: 32 total, 32 up, 32 in 2019-05-20
> 03:38:15.617648 7f1b0a62e700 1 mon.a@0(leader).osd e64459
> e64459: 32 total, 32 up, 32 in
> 2019-05-20 03:38:15.618422 7f1b0a62e700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_write.cc:725
> ] [default] New memtable created with log file: #1788787. Immutable
> memtables: 0.
> 2019-05-20 03:38:15.641025 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:38:15.618497)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu sh.cc:1158] Calling FlushMemTableToOutputFile with column family
> [default], flush slots available 1, compaction slots allowed 1,
> compaction slots scheduled 1 2019-05-20 03:38:15.641035 7f1b0962c700
> 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:264]
> [default] [JOB 57458] Flushing memtable with next logfile: 1788787
> 2019-05-20 03:38:15.641050 7f1b0962c700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316295641042, "job": 57458, "event":
> "flush_started", "num_memtables": 1, "num_entries": 522,
"num_deletes":
> 0, "memory_usage": 32931384}
> 2019-05-20 03:38:15.641054 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:293]
> [default] [JOB 57458] Level-0 flush table #1788788: started 2019-05-20
> 03:38:15.644891 7f1b0a62e700 0 log_channel(cluster) log [DBG] :
> osdmap e64459: 32 total, 32 up, 32 in 2019-05-20 03:38:15.786077
> 7f1b0962c700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316295786051, "cf_name": "default", "job": 57458,
> "event": "table_file_creation", "file_number": 1788788, "file_size":
> 28905705, "table_properties": {"data_size": 28841776, "index_size":
> 5290, "filter_size": 57714, "raw_key_size": 4057,
> "raw_average_key_size": 22, "raw_value_size": 28835274,
> "raw_average_value_size": 161995, "num_data_blocks": 149,
"num_entries":
>
> 178, "filter_policy_name": "rocksdb.BuiltinBloomFilter",
"kDeletedKeys":
>
> "0", "kMergeOperands": "0"}}
> 2019-05-20 03:38:15.786102 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:319]
> [default] [JOB 57458] Level-0 flush table #1788788: 28905705 bytes OK
> 2019-05-20 03:38:15.808704 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:38:15.786117)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:360
> ] [default] Level-0 commit table #1788788 started 2019-05-20
> 03:38:15.808721 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:38:15.808602)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:383
> ] [default] Level-0 commit table #1788788: memtable #1 done 2019-05-20
> 03:38:15.808726 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:38:15.808642) EVENT_LOG_v1 {"time_micros":
> 1558316295808628, "job": 57458, "event": "flush_finished",
"lsm_state":
> [2, 0, 0, 0, 0, 0, 19], "immutable_memtables": 0} 2019-05-20
> 03:38:15.808861 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:38:15.808676)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu sh.cc:132] [default] Level summary: base level 5 max bytes base
> 69194285
>
> files[2 0 0 0 0 0 19] max score 0.50
> 2019-05-20 03:38:15.808894 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388
> ] [JOB 57458] Try to delete WAL files size 28859118, prev total WAL
> file size 28859159, number of live WAL files 2.
> 2019-05-20 03:38:17.014616 7f1b1163c700 0
> mon.a@0(leader).data_health(7108) update_stats avail 91% total
> 50.0GiB, used 4.11GiB, avail 45.9GiB 2019-05-20 03:38:19.736714
> 7f1b0a62e700 1 mon.a@0(leader).osd e64460
> e64460: 32 total, 32 up, 32 in
> 2019-05-20 03:38:19.762553 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64460: 32 total, 32 up, 32 in 2019-05-20
> 03:38:25.640142 7f1b0a62e700 1 mon.a@0(leader).osd e64461
> e64461: 32 total, 32 up, 32 in
> 2019-05-20 03:38:25.666382 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64461: 32 total, 32 up, 32 in 2019-05-20
> 03:38:26.728128 7f1b0a62e700 1 mon.a@0(leader).osd e64462
> e64462: 32 total, 32 up, 32 in
> 2019-05-20 03:38:26.753754 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64462: 32 total, 32 up, 32 in 2019-05-20
> 03:38:27.930914 7f1b0a62e700 1 mon.a@0(leader).osd e64463
> e64463: 32 total, 32 up, 32 in
> 2019-05-20 03:38:27.956420 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64463: 32 total, 32 up, 32 in 2019-05-20
> 03:38:30.271955 7f1b0a62e700 1 mon.a@0(leader).osd e64464
> e64464: 32 total, 32 up, 32 in
> 2019-05-20 03:38:30.297712 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64464: 32 total, 32 up, 32 in 2019-05-20
> 03:38:35.711055 7f1b0a62e700 1 mon.a@0(leader).osd e64465
> e64465: 32 total, 32 up, 32 in
> 2019-05-20 03:38:35.737482 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64465: 32 total, 32 up, 32 in 2019-05-20
> 03:38:36.863809 7f1b0a62e700 1 mon.a@0(leader).osd e64466
> e64466: 32 total, 32 up, 32 in
> 2019-05-20 03:38:36.890569 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64466: 32 total, 32 up, 32 in 2019-05-20
> 03:38:40.090679 7f1b0a62e700 1 mon.a@0(leader).osd e64467
> e64467: 32 total, 32 up, 32 in
> 2019-05-20 03:38:40.116896 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64467: 32 total, 32 up, 32 in 2019-05-20
> 03:38:45.739907 7f1b0a62e700 1 mon.a@0(leader).osd e64468
> e64468: 32 total, 32 up, 32 in
> 2019-05-20 03:38:45.767072 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64468: 32 total, 32 up, 32 in 2019-05-20
> 03:38:46.890384 7f1b0a62e700 1 mon.a@0(leader).osd e64469
> e64469: 32 total, 32 up, 32 in
> 2019-05-20 03:38:46.916813 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64469: 32 total, 32 up, 32 in 2019-05-20
> 03:38:50.117718 7f1b0a62e700 1 mon.a@0(leader).osd e64470
> e64470: 32 total, 32 up, 32 in
> 2019-05-20 03:38:50.144237 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64470: 32 total, 32 up, 32 in 2019-05-20
> 03:38:55.729962 7f1b0a62e700 1 mon.a@0(leader).osd e64471
> e64471: 32 total, 32 up, 32 in
> 2019-05-20 03:38:55.756512 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64471: 32 total, 32 up, 32 in 2019-05-20
> 03:38:56.882171 7f1b0a62e700 1 mon.a@0(leader).osd e64472
> e64472: 32 total, 32 up, 32 in
> 2019-05-20 03:38:56.908626 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64472: 32 total, 32 up, 32 in 2019-05-20
> 03:39:00.138309 7f1b0a62e700 1 mon.a@0(leader).osd e64473
> e64473: 32 total, 32 up, 32 in
> 2019-05-20 03:39:00.164934 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64473: 32 total, 32 up, 32 in 2019-05-20
> 03:39:04.761609 7f1b0a62e700 1 mon.a@0(leader).osd e64474
> e64474: 32 total, 32 up, 32 in
> 2019-05-20 03:39:04.788323 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64474: 32 total, 32 up, 32 in 2019-05-20
> 03:39:09.769143 7f1b0a62e700 1 mon.a@0(leader).osd e64475
> e64475: 32 total, 32 up, 32 in
> 2019-05-20 03:39:09.769992 7f1b0a62e700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_write.cc:725
> ] [default] New memtable created with log file: #1788789. Immutable
> memtables: 0.
> 2019-05-20 03:39:09.792308 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:39:09.770033)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu sh.cc:1158] Calling FlushMemTableToOutputFile with column family
> [default], flush slots available 1, compaction slots allowed 1,
> compaction slots scheduled 1 2019-05-20 03:39:09.792329 7f1b0962c700
> 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:264]
> [default] [JOB 57459] Flushing memtable with next log file: 1788789
> 2019-05-20 03:39:09.792421 7f1b0962c700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316349792351, "job": 57459, "event":
> "flush_started", "num_memtables": 1, "num_entries": 501,
"num_deletes":
> 0, "memory_usage": 32873232}
> 2019-05-20 03:39:09.792427 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:293]
> [default] [JOB 57459] Level-0 flush table #1788790: started 2019-05-20
> 03:39:09.796258 7f1b0a62e700 0 log_channel(cluster) log [DBG] :
> osdmap e64475: 32 total, 32 up, 32 in 2019-05-20 03:39:09.954310
> 7f1b0962c700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1558316349954290, "cf_name": "default", "job": 57459,
> "event": "table_file_creation", "file_number": 1788790, "file_size":
> 28814681, "table_properties": {"data_size": 28751196, "index_size":
> 5076, "filter_size": 57484, "raw_key_size": 3919,
> "raw_average_key_size": 22, "raw_value_size": 28744932,
> "raw_average_value_size": 167121, "num_data_blocks": 143,
"num_entries":
>
> 172, "filter_policy_name": "rocksdb.BuiltinBloomFilter",
"kDeletedKeys":
>
> "0", "kMergeOperands": "0"}}
> 2019-05-20 03:39:09.954338 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:319]
> [default] [JOB 57459] Level-0 flush table #1788790: 28814681 bytes OK
> 2019-05-20 03:39:09.976850 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:39:09.954351)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:360
> ] [default] Level-0 commit table #1788790 started 2019-05-20
> 03:39:09.976866 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:39:09.976744)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:383
> ] [default] Level-0 commit table #1788790: memtable #1 done 2019-05-20
> 03:39:09.976872 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:39:09.976783) EVENT_LOG_v1 {"time_micros":
> 1558316349976770, "job": 57459, "event": "flush_finished",
"lsm_state":
> [3, 0, 0, 0, 0, 0, 19], "immutable_memtables": 0} 2019-05-20
> 03:39:09.976877 7f1b0962c700 4 rocksdb: (Original Log Time
> 2019/05/20-03:39:09.976815)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f
> lu sh.cc:132] [default] Level summary: base level 5 max bytes base
> 69194285
>
> files[3 0 0 0 0 0 19] max score 0.75
> 2019-05-20 03:39:09.976894 7f1b0962c700 4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_
> AR
> CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas
> e/
> 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388
> ] [JOB 57459] Try to delete WAL files size 28767628, prev total WAL
> file size 28767669, number of live WAL files 2.
> 2019-05-20 03:39:10.921941 7f1b0a62e700 1 mon.a@0(leader).osd e64476
> e64476: 32 total, 32 up, 32 in
> 2019-05-20 03:39:10.948342 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64476: 32 total, 32 up, 32 in 2019-05-20
> 03:39:14.782433 7f1b0a62e700 1 mon.a@0(leader).osd e64477
> e64477: 32 total, 32 up, 32 in
> 2019-05-20 03:39:14.809139 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64477: 32 total, 32 up, 32 in 2019-05-20
> 03:39:17.014951 7f1b1163c700 0
> mon.a@0(leader).data_health(7108) update_stats avail 91% total
> 50.0GiB, used 4.13GiB, avail 45.9GiB 2019-05-20 03:39:19.801742
> 7f1b0a62e700 1 mon.a@0(leader).osd e64478
> e64478: 32 total, 32 up, 32 in
> 2019-05-20 03:39:19.828970 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64478: 32 total, 32 up, 32 in 2019-05-20
> 03:39:24.747871 7f1b0a62e700 1 mon.a@0(leader).osd e64479
> e64479: 32 total, 32 up, 32 in
> 2019-05-20 03:39:24.775203 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64479: 32 total, 32 up, 32 in 2019-05-20
> 03:39:29.831760 7f1b0a62e700 1 mon.a@0(leader).osd e64480
> e64480: 32 total, 32 up, 32 in
> 2019-05-20 03:39:29.859184 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64480: 32 total, 32 up, 32 in 2019-05-20
> 03:39:34.747989 7f1b0a62e700 1 mon.a@0(leader).osd e64481
> e64481: 32 total, 32 up, 32 in
> 2019-05-20 03:39:34.775166 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64481: 32 total, 32 up, 32 in 2019-05-20
> 03:39:39.863809 7f1b0a62e700 1 mon.a@0(leader).osd e64482
> e64482: 32 total, 32 up, 32 in
> 2019-05-20 03:39:39.891227 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64482: 32 total, 32 up, 32 in 2019-05-20
> 03:39:44.750833 7f1b0a62e700 1 mon.a@0(leader).osd e64483
> e64483: 32 total, 32 up, 32 in
> 2019-05-20 03:39:44.778742 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64483: 32 total, 32 up, 32 in 2019-05-20
> 03:39:49.900624 7f1b0a62e700 1 mon.a@0(leader).osd e64484
> e64484: 32 total, 32 up, 32 in
> 2019-05-20 03:39:49.928412 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64484: 32 total, 32 up, 32 in 2019-05-20
> 03:39:54.743844 7f1b0a62e700 1 mon.a@0(leader).osd e64485
> e64485: 32 total, 32 up, 32 in
> 2019-05-20 03:39:54.773021 7f1b0a62e700 0 log_channel(cluster) log
> [DBG] : osdmap e64485: 32 total, 32 up, 32 in 2019-05-20
> 03:39:59.937640 7f1b0a62e700 1 mon.a@0(leader).osd e64486
> e64486: 32 total, 32 up, 32 in
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
