[ceph-users] Re: another osd_pglog memory usage incident

2020-10-07 Thread Wido den Hollander
On 07/10/2020 14:08, Dan van der Ster wrote: Hi all, This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean). After readi

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-07 Thread Dan van der Ster
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander wrote: > > > > On 07/10/2020 14:08, Dan van der Ster wrote: > > Hi all, > > > > This morning some osds in our S3 cluster started going OOM, after > > restarting them I noticed that the osd_pglog is using >1.5GB per osd. > > (This is on an osd with

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-07 Thread Wido den Hollander
On 07/10/2020 16:00, Dan van der Ster wrote: On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander wrote: On 07/10/2020 14:08, Dan van der Ster wrote: Hi all, This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per o

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-09 Thread Harald Staub
On 07.10.20 21:00, Wido den Hollander wrote: On 07/10/2020 16:00, Dan van der Ster wrote: On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander wrote: On 07/10/2020 14:08, Dan van der Ster wrote: Hi all, This morning some osds in our S3 cluster started going OOM, after restarting them I not

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-09 Thread Dan van der Ster
On Fri, Oct 9, 2020 at 1:42 PM Harald Staub wrote: > > On 07.10.20 21:00, Wido den Hollander wrote: > > > > > > On 07/10/2020 16:00, Dan van der Ster wrote: > >> On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander wrote: > >>> > >>> > >>> > >>> On 07/10/2020 14:08, Dan van der Ster wrote: > Hi

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-09 Thread Harald Staub
On 09.10.20 13:55, Dan van der Ster wrote: [...] I also noticed a possible relationship with scrubbing -- One week ago we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I wonder if the increased read/write ratio somehow led to an exploding buffer_anon. Do things stabilize on your

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-09 Thread Marc Roos
-Original Message- Cc: ceph-users Subject: [ceph-users] Re: another osd_pglog memory usage incident On 09.10.20 13:55, Dan van der Ster wrote: [...] > I also noticed a possible relationship with scrubbing -- One week ago > we increased to osd_max_scrubs=5 to clear out a scrubbing back

[ceph-users] Re: another osd_pglog memory usage incident

2020-10-09 Thread Dan van der Ster
iggered this > >today). > > How can I check how much ram my pg_logs are using? ceph daemon osd.x dump_mempools | jq .mempool.by_pool.osd_pglog > > > > -Original Message- > Cc: ceph-users > Subject: [ceph-users] Re: another osd_pglog memory usage incident > &g