On Fri, Sep 2, 2016 at 5:10 PM, Sean Redmond <sean.redmo...@gmail.com> wrote: > I have checked all the servers in scope running 'dmesg | grep -i stale' and > it does not yield any results. > > As a test I have rebooted the servers in scope and I can still replicate the > behavior 100% of the time. >
Can you reproduce this bug manually? (updating file on one server and reading the file on another server). If you do, please enable debug_mds=10 , repeat the steps that reproduce this and send log to us. Regards Yan, Zheng > On Fri, Sep 2, 2016 at 4:37 AM, Yan, Zheng <uker...@gmail.com> wrote: >> >> I think about this again. This issue could be caused by stale session. >> Could you check kernel logs of your servers. Are there any ceph >> related kernel message (such as "ceph: mds0 caps stale") >> >> Regards >> Yan, Zheng >> >> >> On Thu, Sep 1, 2016 at 11:02 PM, Sean Redmond <sean.redmo...@gmail.com> >> wrote: >> > Hi, >> > >> > It seems to be using syscall mmap() from what I read this indicates it >> > is >> > using memory-mapped IO. >> > >> > Please see a strace here: http://pastebin.com/6wjhSNrP >> > >> > Thanks >> > >> > On Wed, Aug 31, 2016 at 5:51 PM, Sean Redmond <sean.redmo...@gmail.com> >> > wrote: >> >> >> >> I am not sure how to tell? >> >> >> >> Server1 and Server2 mount the ceph file system using kernel client >> >> 4.7.2 >> >> and I can replicate the problem using '/usr/bin/sum' to read the file >> >> or a >> >> http GET request via a web server (apache). >> >> >> >> On Wed, Aug 31, 2016 at 2:38 PM, Yan, Zheng <uker...@gmail.com> wrote: >> >>> >> >>> On Wed, Aug 31, 2016 at 12:49 AM, Sean Redmond >> >>> <sean.redmo...@gmail.com> >> >>> wrote: >> >>> > Hi, >> >>> > >> >>> > I have been able to pick through the process a little further and >> >>> > replicate >> >>> > it via the command line. The flow seems looks like this: >> >>> > >> >>> > 1) The user uploads an image to webserver server 'uploader01' it >> >>> > gets >> >>> > written to a path such as >> >>> > '/cephfs/webdata/static/456/JHL/66448H-755h.jpg' >> >>> > on cephfs >> >>> > >> >>> > 2) The MDS makes the file meta data available for this new file >> >>> > immediately >> >>> > to all clients. >> >>> > >> >>> > 3) The 'uploader01' server asynchronously commits the file contents >> >>> > to >> >>> > disk >> >>> > as sync is not explicitly called during the upload. >> >>> > >> >>> > 4) Before step 3 is done the visitor requests the file via one of >> >>> > two >> >>> > web >> >>> > servers server1 or server2 - the MDS provides the meta data but the >> >>> > contents >> >>> > of the file is not committed to disk yet so the data read returns >> >>> > 0's - >> >>> > This >> >>> > is then cached by the file system page cache until it expires or is >> >>> > flushed >> >>> > manually. >> >>> >> >>> do server1 or server2 use memory-mapped IO to read the file? >> >>> >> >>> Regards >> >>> Yan, Zheng >> >>> >> >>> > >> >>> > 5) As step 4 typically only happens on one of the two web servers >> >>> > before >> >>> > step 3 is complete we get the mismatch between server1 and server2 >> >>> > file >> >>> > system page cache. >> >>> > >> >>> > The below demonstrates how to reproduce this issue >> >>> > >> >>> > http://pastebin.com/QK8AemAb >> >>> > >> >>> > As we can see the checksum of the file returned by the web server is >> >>> > 0 >> >>> > as >> >>> > the file contents has not been flushed to disk from server >> >>> > uploader01 >> >>> > >> >>> > If however we call ‘sync’ as shown below the checksum is correct: >> >>> > >> >>> > http://pastebin.com/p4CfhEFt >> >>> > >> >>> > If we also wait for 10 seconds for the kernel to flush the dirty >> >>> > pages, >> >>> > we >> >>> > can also see the checksum is valid: >> >>> > >> >>> > http://pastebin.com/1w6UZzNQ >> >>> > >> >>> > It looks it maybe a race between the time it takes the uploader01 >> >>> > server to >> >>> > commit the file to the file system and the fast incoming read >> >>> > request >> >>> > from >> >>> > the visiting user to server1 or server2. >> >>> > >> >>> > Thanks >> >>> > >> >>> > >> >>> > On Tue, Aug 30, 2016 at 10:21 AM, Sean Redmond >> >>> > <sean.redmo...@gmail.com> >> >>> > wrote: >> >>> >> >> >>> >> You are correct it only seems to impact recently modified files. >> >>> >> >> >>> >> On Tue, Aug 30, 2016 at 3:36 AM, Yan, Zheng <uker...@gmail.com> >> >>> >> wrote: >> >>> >>> >> >>> >>> On Tue, Aug 30, 2016 at 2:11 AM, Gregory Farnum >> >>> >>> <gfar...@redhat.com> >> >>> >>> wrote: >> >>> >>> > On Mon, Aug 29, 2016 at 7:14 AM, Sean Redmond >> >>> >>> > <sean.redmo...@gmail.com> >> >>> >>> > wrote: >> >>> >>> >> Hi, >> >>> >>> >> >> >>> >>> >> I am running cephfs (10.2.2) with kernel 4.7.0-1. I have >> >>> >>> >> noticed >> >>> >>> >> that >> >>> >>> >> frequently static files are showing empty when serviced via a >> >>> >>> >> web >> >>> >>> >> server >> >>> >>> >> (apache). I have tracked this down further and can see when >> >>> >>> >> running a >> >>> >>> >> checksum against the file on the cephfs file system on the node >> >>> >>> >> serving the >> >>> >>> >> empty http response the checksum is '00000' >> >>> >>> >> >> >>> >>> >> The below shows the checksum on a defective node. >> >>> >>> >> >> >>> >>> >> [root@server2]# ls -al >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> -rw-r--r-- 1 apache apache 53317 Aug 28 23:46 >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> >>> >>> It seems this file was modified recently. Maybe the web server >> >>> >>> silently modifies the files. Please check if this issue happens on >> >>> >>> older files. >> >>> >>> >> >>> >>> Regards >> >>> >>> Yan, Zheng >> >>> >>> >> >>> >>> >> >> >>> >>> >> [root@server2]# sum >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> 00000 53 >> >>> >>> > >> >>> >>> > So can we presume there are no file contents, and it's just 53 >> >>> >>> > blocks >> >>> >>> > of zeros? >> >>> >>> > >> >>> >>> > This doesn't sound familiar to me; Zheng, do you have any ideas? >> >>> >>> > Anyway, ceph-fuse shouldn't be susceptible to this bug even with >> >>> >>> > the >> >>> >>> > page cache enabled; if you're just serving stuff via the web >> >>> >>> > it's >> >>> >>> > probably a better idea anyway (harder to break, easier to >> >>> >>> > update, >> >>> >>> > etc). >> >>> >>> > -Greg >> >>> >>> > >> >>> >>> >> >> >>> >>> >> The below shows the checksum on a working node. >> >>> >>> >> >> >>> >>> >> [root@server1]# ls -al >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> -rw-r--r-- 1 apache apache 53317 Aug 28 23:46 >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> >> >>> >>> >> [root@server1]# sum >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> 03620 53 >> >>> >>> >> [root@server1]# >> >>> >>> >> >> >>> >>> >> If I flush the cache as shown below the checksum returns as >> >>> >>> >> expected >> >>> >>> >> and the >> >>> >>> >> web server serves up valid content. >> >>> >>> >> >> >>> >>> >> [root@server2]# echo 3 > /proc/sys/vm/drop_caches >> >>> >>> >> [root@server2]# sum >> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg >> >>> >>> >> 03620 53 >> >>> >>> >> >> >>> >>> >> After some time typically less than 1hr the issue repeats, It >> >>> >>> >> seems to >> >>> >>> >> not >> >>> >>> >> repeat if I take any one of the servers out of the LB and only >> >>> >>> >> serve >> >>> >>> >> requests from one of the servers. >> >>> >>> >> >> >>> >>> >> I may try and use the FUSE client has has a mount option >> >>> >>> >> direct_io >> >>> >>> >> that >> >>> >>> >> looks to disable page cache. >> >>> >>> >> >> >>> >>> >> I have been hunting in the ML and tracker but could not see >> >>> >>> >> anything >> >>> >>> >> really >> >>> >>> >> close to this issue, Any input or feedback on similar >> >>> >>> >> experiences >> >>> >>> >> is >> >>> >>> >> welcome. >> >>> >>> >> >> >>> >>> >> Thanks >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> _______________________________________________ >> >>> >>> >> ceph-users mailing list >> >>> >>> >> ceph-users@lists.ceph.com >> >>> >>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>> >>> >> >> >>> >>> > _______________________________________________ >> >>> >>> > ceph-users mailing list >> >>> >>> > ceph-users@lists.ceph.com >> >>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>> >> >> >>> >> >> >>> > >> >> >> >> >> > > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
