On Fri, Jan 15, 2016 at 3:28 AM, Mike Carlson <m...@bayphoto.com> wrote: > Thank you for the reply Zheng > > We tried set mds bal frag to true, but the end result was less than > desirable. All nfs and smb clients could no longer browse the share, they > would hang on a directory with anything more than a few hundred files. > > We then tried to back out the active/active mds change, no luck, stopping > one of the mds's (mds 1) prevented us from mounting the cephfs filesystem > > So we failed and removed the secondary MDS, and now our primary mds is stuck > in a "resovle" state: > > # ceph -s > cluster cabd1728-2eca-4e18-a581-b4885364e5a4 > health HEALTH_WARN > clock skew detected on mon.lts-mon > mds cluster is degraded > Monitor clock skew detected > monmap e1: 4 mons at > {lts-mon=10.5.68.236:6789/0,lts-osd1=10.5.68.229:6789/0,lts-osd2=10.5.68.230:6789/0,lts-osd3=10.5.68.203:6789/0} > election epoch 1282, quorum 0,1,2,3 > lts-osd3,lts-osd1,lts-osd2,lts-mon > mdsmap e7892: 1/2/1 up {0=lts-mon=up:resolve} > osdmap e10183: 102 osds: 101 up, 101 in > pgmap v6714309: 4192 pgs, 7 pools, 31748 GB data, 23494 kobjects > 96188 GB used, 273 TB / 367 TB avail > 4188 active+clean > 4 active+clean+scrubbing+deep > > Now we are really down for the count. We cannot get our MDS back up in an > active state and none of our data is accessible.
you can't remove active mds this way, you need to: 1. make sure all active mds are running 2. run 'ceph mds set max_mds 1' 3. run 'ceph mds stop 1' step 3 changes the second mds's state to stopping. Wait a while, the second mds will go to standby state. Occasionally, the second MDS can stuck in stopping state. If it happens, restart all MDS, then repeat step 3. Regards Yan, Zheng > > > On Wed, Jan 13, 2016 at 7:05 PM, Yan, Zheng <uker...@gmail.com> wrote: >> >> On Thu, Jan 14, 2016 at 3:37 AM, Mike Carlson <m...@bayphoto.com> wrote: >> > Hey Greg, >> > >> > The inconsistent view is only over nfs/smb on top of our /ceph mount. >> > >> > When I look directly on the /ceph mount (which is using the cephfs >> > kernel >> > module), everything looks fine >> > >> > It is possible that this issue just went unnoticed, and it only being a >> > infernalis problem is just a red herring. With that, it is oddly >> > coincidental that we just started seeing issues. >> >> This seems like seekdir bugs in kernel client, could you try 4.0+ kernel. >> >> Besides, do you enable "mds bal frag" for ceph-mds >> >> >> Regards >> Yan, Zheng >> >> >> >> > >> > On Wed, Jan 13, 2016 at 11:30 AM, Gregory Farnum <gfar...@redhat.com> >> > wrote: >> >> >> >> On Wed, Jan 13, 2016 at 11:24 AM, Mike Carlson <m...@bayphoto.com> >> >> wrote: >> >> > Hello. >> >> > >> >> > Since we upgraded to Infernalis last, we have noticed a severe >> >> > problem >> >> > with >> >> > cephfs when we have it shared over Samba and NFS >> >> > >> >> > Directory listings are showing an inconsistent view of the files: >> >> > >> >> > >> >> > $ ls /lts-mon/BD/xmlExport/ | wc -l >> >> > 100 >> >> > $ sudo umount /lts-mon >> >> > $ sudo mount /lts-mon >> >> > $ ls /lts-mon/BD/xmlExport/ | wc -l >> >> > 3507 >> >> > >> >> > >> >> > The only work around I have found is un-mounting and re-mounting the >> >> > nfs >> >> > share, that seems to clear it up >> >> > Same with samba, I'd post it here but its thousands of lines. I can >> >> > add >> >> > additional details on request. >> >> > >> >> > This happened after our upgrade to infernalis. Is it possible the MDS >> >> > is >> >> > in >> >> > an inconsistent state? >> >> >> >> So this didn't happen to you until after you upgraded? Are you seeing >> >> missing files when looking at cephfs directly, or only over the >> >> NFS/Samba re-exports? Are you also sharing Samba by re-exporting the >> >> kernel cephfs mount? >> >> >> >> Zheng, any ideas about kernel issues which might cause this or be more >> >> visible under infernalis? >> >> -Greg >> >> >> >> > >> >> > We have cephfs mounted on a server using the built in cephfs kernel >> >> > module: >> >> > >> >> > lts-mon:6789:/ /ceph ceph >> >> > name=admin,secretfile=/etc/ceph/admin.secret,noauto,_netdev >> >> > >> >> > >> >> > We are running all of our ceph nodes on ubuntu 14.04 LTS. Samba is up >> >> > to >> >> > date, 4.1.6, and we export nfsv3 to linux and freebsd systems. All >> >> > seem >> >> > to >> >> > exhibit the same behavior. >> >> > >> >> > system info: >> >> > >> >> > # uname -a >> >> > Linux lts-osd1 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 >> >> > UTC >> >> > 2015 x86_64 x86_64 x86_64 GNU/Linux >> >> > root@lts-osd1:~# lsb >> >> > lsblk lsb_release >> >> > root@lts-osd1:~# lsb_release -a >> >> > No LSB modules are available. >> >> > Distributor ID: Ubuntu >> >> > Description: Ubuntu 14.04.3 LTS >> >> > Release: 14.04 >> >> > Codename: trusty >> >> > >> >> > >> >> > package info: >> >> > >> >> > # dpkg -l|grep ceph >> >> > ii ceph 9.2.0-1trusty >> >> > amd64 distributed storage and file system >> >> > ii ceph-common 9.2.0-1trusty >> >> > amd64 common utilities to mount and interact with a ceph >> >> > storage >> >> > cluster >> >> > ii ceph-fs-common 9.2.0-1trusty >> >> > amd64 common utilities to mount and interact with a ceph file >> >> > system >> >> > ii ceph-mds 9.2.0-1trusty >> >> > amd64 metadata server for the ceph distributed file system >> >> > ii libcephfs1 9.2.0-1trusty >> >> > amd64 Ceph distributed file system client library >> >> > ii python-ceph 9.2.0-1trusty >> >> > amd64 Meta-package for python libraries for the Ceph libraries >> >> > ii python-cephfs 9.2.0-1trusty >> >> > amd64 Python libraries for the Ceph libcephfs library >> >> > >> >> > >> >> > What is interesting, is a directory or file will not show up in a >> >> > listing, >> >> > however, if we directly access the file, it shows up in that >> >> > instance: >> >> > >> >> > >> >> > # ls -al |grep SCHOOL >> >> > # ls -alnd SCHOOL667055 >> >> > drwxrwsr-x 1 21695 21183 2962751438 Jan 13 09:33 SCHOOL667055 >> >> > >> >> > >> >> > Any tips are appreciated! >> >> > >> >> > Thanks, >> >> > Mike C >> >> > >> >> > >> >> > _______________________________________________ >> >> > ceph-users mailing list >> >> > ceph-users@lists.ceph.com >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >> > >> > >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com