I'm assuming you use RGW and that you have a GC pool for RGW. It also might beat assumed that your GC pool only has 8 PGs. Are any of those guesses correct?
On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski <[email protected]> wrote: > Issue tracker http://tracker.ceph.com/issues/23801. > Still don't know why only particular OSDs write this information to log > files. > > Jakub > > On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski <[email protected]> > wrote: > >> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage >> collection objects to process >> >> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key=" >> entries per OSD log file >> hostA: >> /var/log/ceph/ceph-osd.58.log >> 1826467 >> hostB: >> /var/log/ceph/ceph-osd.88.log >> 2924241 >> hostC: >> /var/log/ceph/ceph-osd.153.log >> 581002 >> /var/log/ceph/ceph-osd.164.log >> 3278606 >> hostD: >> /var/log/ceph/ceph-osd.95.log >> 1426963 >> hostE: >> /var/log/ceph/ceph-osd.4.log >> 2716914 >> /var/log/ceph/ceph-osd.53.log >> 943749 >> hostF: >> /var/log/ceph/ceph-osd.172.log >> 4085334 >> >> >> # radosgw-admin gc list --include-all|grep oid |wc -l >> 302357 >> # >> >> Can anyone please explain what is going on ? >> >> Thanks! >> Jakub >> >> On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski <[email protected]> >> wrote: >> >>> Hi, >>> >>> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of >>> records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the >>> corresponding log files, e.g. >>> >>> 2018-08-07 04:34:06.000585 7fdd8f012700 0 <cls> >>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries >>> end_key=1_01533616446.000580407 >>> 2018-08-07 04:34:06.001888 7fdd8f012700 0 <cls> >>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries >>> end_key=1_01533616446.001886318 >>> 2018-08-07 04:34:06.003395 7fdd8f012700 0 <cls> >>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries >>> end_key=1_01533616446.003390299 >>> 2018-08-07 04:34:06.005205 7fdd8f012700 0 <cls> >>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries >>> end_key=1_01533616446.005200341 >>> >>> # grep '2018-08-07 04:34:06' /var/log/ceph/ceph-osd.4.log |wc -l >>> 712 >>> # >>> >>> At the same time there were like 500 000 expired garbage collection >>> objects. >>> >>> Log level of OSD subsystem is set to default 1/5 across all OSDs. >>> >>> I wonder why only few OSDs record this information and is it something >>> to be logged in log level = 1 or maybe higher? >>> https://github.com/ceph/ceph/blob/v12.2.5/src/cls/rgw/cls_rgw.cc#L3284 >>> >>> Thanks >>> Jakub >>> >> _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
