Re: [ceph-users] cephfs page cache

Hi, Thanks for looking into this, this seems to mitigate the problem. Do you think this is just related to httpd or is going to impact other services such as nginx and be a wider point to know about / document? Thanks On Mon, Oct 24, 2016 at 2:28 PM, Yan, Zheng wrote: > I

Re: [ceph-users] cephfs page cache

I finally reproduced this issue. Adding following lines to httpd.conf can workaround this issue. EnableMMAP off EnableSendfile off On Sat, Sep 3, 2016 at 11:07 AM, Yan, Zheng wrote: > On Fri, Sep 2, 2016 at 5:10 PM, Sean Redmond wrote: >> I have

Re: [ceph-users] cephfs page cache

On Fri, Sep 2, 2016 at 5:10 PM, Sean Redmond wrote: > I have checked all the servers in scope running 'dmesg | grep -i stale' and > it does not yield any results. > > As a test I have rebooted the servers in scope and I can still replicate the > behavior 100% of the time.

Re: [ceph-users] cephfs page cache

Hi, The work around is fine, I still think there could be a bug here. I will try and spend some time in the next few days to write something to test with Regular buffered IO. But even using memory-mapped IO I would not expect the read request from server1 or server2 to get zeros's in place of

Re: [ceph-users] cephfs page cache

On Fri, Sep 2, 2016 at 11:35 AM, Sean Redmond wrote: > Hi, > > That makes sense, I have worked around this by forcing the sync within the > application running under apache and it is working very well now without the > need for the 'sync' mount option. > > What

Re: [ceph-users] cephfs page cache

Hi, That makes sense, I have worked around this by forcing the sync within the application running under apache and it is working very well now without the need for the 'sync' mount option. What interesting is that in the pastebin provided below it shows a way to replicate this, I was just using

Re: [ceph-users] cephfs page cache

On Thu, Sep 1, 2016 at 8:02 AM, Sean Redmond wrote: > Hi, > > It seems to be using syscall mmap() from what I read this indicates it is > using memory-mapped IO. > > Please see a strace here: http://pastebin.com/6wjhSNrP Zheng meant is Apache using memory-mapped IO. From

Re: [ceph-users] cephfs page cache

I have checked all the servers in scope running 'dmesg | grep -i stale' and it does not yield any results. As a test I have rebooted the servers in scope and I can still replicate the behavior 100% of the time. On Fri, Sep 2, 2016 at 4:37 AM, Yan, Zheng wrote: > I think

Re: [ceph-users] cephfs page cache

I think about this again. This issue could be caused by stale session. Could you check kernel logs of your servers. Are there any ceph related kernel message (such as "ceph: mds0 caps stale") Regards Yan, Zheng On Thu, Sep 1, 2016 at 11:02 PM, Sean Redmond wrote: > Hi,

Re: [ceph-users] cephfs page cache

Hi, It seems to be using syscall mmap() from what I read this indicates it is using memory-mapped IO. Please see a strace here: http://pastebin.com/6wjhSNrP Thanks On Wed, Aug 31, 2016 at 5:51 PM, Sean Redmond wrote: > I am not sure how to tell? > > Server1 and

Re: [ceph-users] cephfs page cache

I am not sure how to tell? Server1 and Server2 mount the ceph file system using kernel client 4.7.2 and I can replicate the problem using '/usr/bin/sum' to read the file or a http GET request via a web server (apache). On Wed, Aug 31, 2016 at 2:38 PM, Yan, Zheng wrote: > On

Re: [ceph-users] cephfs page cache

On Wed, Aug 31, 2016 at 12:49 AM, Sean Redmond wrote: > Hi, > > I have been able to pick through the process a little further and replicate > it via the command line. The flow seems looks like this: > > 1) The user uploads an image to webserver server 'uploader01' it gets

Re: [ceph-users] cephfs page cache

It seems using the 'sync' mount option on the server uploader01 is also a valid work around. Is it a problem that the meta data is available to other cephfs clients ahead of the file contents being flushed by the client doing the write? I think having an invalid page cache of zeros is a problem

Re: [ceph-users] cephfs page cache

Hi, I have been able to pick through the process a little further and replicate it via the command line. The flow seems looks like this: 1) The user uploads an image to webserver server 'uploader01' it gets written to a path such as '/cephfs/webdata/static/456/JHL/66448H-755h.jpg' on cephfs 2)

Re: [ceph-users] cephfs page cache

You are correct it only seems to impact recently modified files. On Tue, Aug 30, 2016 at 3:36 AM, Yan, Zheng wrote: > On Tue, Aug 30, 2016 at 2:11 AM, Gregory Farnum > wrote: > > On Mon, Aug 29, 2016 at 7:14 AM, Sean Redmond >

Re: [ceph-users] cephfs page cache

On Tue, Aug 30, 2016 at 2:11 AM, Gregory Farnum wrote: > On Mon, Aug 29, 2016 at 7:14 AM, Sean Redmond wrote: >> Hi, >> >> I am running cephfs (10.2.2) with kernel 4.7.0-1. I have noticed that >> frequently static files are showing empty when serviced

Re: [ceph-users] cephfs page cache

Hi, Yes the file has no contents until the page cache is flushed. I will give the fuse client a try and report back. Thanks On Mon, Aug 29, 2016 at 7:11 PM, Gregory Farnum wrote: > On Mon, Aug 29, 2016 at 7:14 AM, Sean Redmond > wrote: > > Hi, >

Re: [ceph-users] cephfs page cache

On Mon, Aug 29, 2016 at 7:14 AM, Sean Redmond wrote: > Hi, > > I am running cephfs (10.2.2) with kernel 4.7.0-1. I have noticed that > frequently static files are showing empty when serviced via a web server > (apache). I have tracked this down further and can see when

[ceph-users] cephfs page cache

Hi, I am running cephfs (10.2.2) with kernel 4.7.0-1. I have noticed that frequently static files are showing empty when serviced via a web server (apache). I have tracked this down further and can see when running a checksum against the file on the cephfs file system on the node serving the

[ceph-users] cephfs page cache

Hi, I am running cephfs (10.2.2) with kernel 4.7.0-1. I have noticed that frequently static files are showing empty when serviced via a web server (apache). I have tracked this down further and can see when running a checksum against the file on the cephfs file system on the node serving the