Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-12-16 Thread jesper
> If a CephFS client receive a cap release request and it is able to > perform it (no processes accessing the file at the moment), the client > cleaned up its internal state and allows the MDS to release the cap. > This cleanup also involves removing file data from the page cache. > > If your MDS w

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-11-03 Thread Burkhard Linke
Hi, On 03.11.18 10:31, jes...@krogh.cc wrote: I suspect that mds asked client to trim its cache. Please run following commands on an idle client. In the mean time - we migrated to the RH Ceph version and deliered the MDS both SSD's and more memory and the problem went away. It still puzzles my

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-11-03 Thread jesper
> I suspect that mds asked client to trim its cache. Please run > following commands on an idle client. In the mean time - we migrated to the RH Ceph version and deliered the MDS both SSD's and more memory and the problem went away. It still puzzles my mind a bit - why is there a connection betwe

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-18 Thread Yan, Zheng
On Mon, Oct 15, 2018 at 9:54 PM Dietmar Rieder wrote: > > On 10/15/18 1:17 PM, jes...@krogh.cc wrote: > >> On 10/15/18 12:41 PM, Dietmar Rieder wrote: > >>> No big difference here. > >>> all CentOS 7.5 official kernel 3.10.0-862.11.6.el7.x86_64 > >> > >> ...forgot to mention: all is luminous ceph-

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-15 Thread Dietmar Rieder
On 10/15/18 1:17 PM, jes...@krogh.cc wrote: >> On 10/15/18 12:41 PM, Dietmar Rieder wrote: >>> No big difference here. >>> all CentOS 7.5 official kernel 3.10.0-862.11.6.el7.x86_64 >> >> ...forgot to mention: all is luminous ceph-12.2.7 > > Thanks for your time in testing, this is very valueable t

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-15 Thread jesper
> On 10/15/18 12:41 PM, Dietmar Rieder wrote: >> No big difference here. >> all CentOS 7.5 official kernel 3.10.0-862.11.6.el7.x86_64 > > ...forgot to mention: all is luminous ceph-12.2.7 Thanks for your time in testing, this is very valueable to me in the debugging. 2 questions: Did you "sleep 9

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-15 Thread Dietmar Rieder
On 10/15/18 12:41 PM, Dietmar Rieder wrote: > On 10/15/18 12:02 PM, jes...@krogh.cc wrote: On Sun, Oct 14, 2018 at 8:21 PM wrote: how many cephfs mounts that access the file? Is is possible that some program opens that file in RW mode (even they just read the file)? >>> >>> >>> The

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-15 Thread Dietmar Rieder
On 10/15/18 12:02 PM, jes...@krogh.cc wrote: >>> On Sun, Oct 14, 2018 at 8:21 PM wrote: >>> how many cephfs mounts that access the file? Is is possible that some >>> program opens that file in RW mode (even they just read the file)? >> >> >> The nature of the program is that it is "prepped" by one

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-15 Thread jesper
>> On Sun, Oct 14, 2018 at 8:21 PM wrote: >> how many cephfs mounts that access the file? Is is possible that some >> program opens that file in RW mode (even they just read the file)? > > > The nature of the program is that it is "prepped" by one-set of commands > and queried by another, thus the

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread jesper
> On Sun, Oct 14, 2018 at 8:21 PM wrote: > how many cephfs mounts that access the file? Is is possible that some > program opens that file in RW mode (even they just read the file)? The nature of the program is that it is "prepped" by one-set of commands and queried by another, thus the RW case

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread Yan, Zheng
On Sun, Oct 14, 2018 at 8:21 PM wrote: > > Hi > > We have a dataset of ~300 GB on CephFS which as being used for computations > over and over agian .. being refreshed daily or similar. > > When hosting it on NFS after refresh, they are transferred, but from > there - they would be sitting in the k

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread jesper
> Actual amount of memory used by VFS cache is available through 'grep > Cached /proc/meminfo'. slabtop provides information about cache > of inodes, dentries, and IO memory buffers (buffer_head). Thanks, that was also what I got out of it. And why I reported "free" output in the first as it also

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread Sergey Malinin
Actual amount of memory used by VFS cache is available through 'grep Cached /proc/meminfo'. slabtop provides information about cache of inodes, dentries, and IO memory buffers (buffer_head). > On 14.10.2018, at 17:28, jes...@krogh.cc wrote: > >> Try looking in /proc/slabinfo / slabtop during y

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread jesper
> Try looking in /proc/slabinfo / slabtop during your tests. I need a bit of guidance here.. Does the slabinfo cover the VFS page cache ? .. I cannot seem to find any traces (sorting by size on machines with a huge cache does not really give anything). Perhaps I'm holding the screwdriver wrong?

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread Sergey Malinin
Try looking in /proc/slabinfo / slabtop during your tests. > On 14.10.2018, at 15:21, jes...@krogh.cc wrote: > > Hi > > We have a dataset of ~300 GB on CephFS which as being used for computations > over and over agian .. being refreshed daily or similar. > > When hosting it on NFS after refres

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread Jesper Krogh
On 14 Oct 2018, at 15.26, John Hearns wrote: > > This is a general question for the ceph list. > Should Jesper be looking at these vm tunables? > vm.dirty_ratio > vm.dirty_centisecs > > What effect do they have when using Cephfs? This situation is a read only, thus no dirty data in page cache.

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread John Hearns
This is a general question for the ceph list. Should Jesper be looking at these vm tunables? vm.dirty_ratio vm.dirty_centisecs What effect do they have when using Cephfs? On Sun, 14 Oct 2018 at 14:24, John Hearns wrote: > Hej Jesper. > Sorry I do not have a direct answer to your question. > Whe

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread John Hearns
Hej Jesper. Sorry I do not have a direct answer to your question. When looking at memory usage, I often use this command: watch cat /rpoc/meminfo On Sun, 14 Oct 2018 at 13:22, wrote: > Hi > > We have a dataset of ~300 GB on CephFS which as being used for computations > over and over agian

[ceph-users] cephfs kernel client - page cache being invaildated.

2018-10-14 Thread jesper
Hi We have a dataset of ~300 GB on CephFS which as being used for computations over and over agian .. being refreshed daily or similar. When hosting it on NFS after refresh, they are transferred, but from there - they would be sitting in the kernel page cache of the client until they are refreshe