Sure Start of upgrade: 15:36 Start of issue: 21:51
On Mon, Aug 22, 2016 at 9:15 PM, Atin Mukherjee <amukh...@redhat.com> wrote: > > > On Tue, Aug 23, 2016 at 4:17 AM, Steve Dainard <sdain...@spd1.com> wrote: > >> About 5 hours after upgrading gluster 3.7.6 -> 3.7.13 on Centos 7, one of >> my gluster servers disconnected its volume. The other two volumes this host >> serves were not affected. >> >> # gluster volume status storage >> Status of volume: storage >> Gluster process TCP Port RDMA Port Online >> Pid >> ------------------------------------------------------------ >> ------------------ >> Brick 10.0.231.50:/mnt/raid6-storage/storage >> 49159 0 Y 30743 >> Brick 10.0.231.51:/mnt/raid6-storage/storage >> 49159 0 Y 676 >> *Brick 10.0.231.52:/mnt/raid6-storage/storag**e >> N/A N/A N N/A * >> Brick 10.0.231.53:/mnt/raid6-storage/storage >> 49154 0 Y 10253 >> Brick 10.0.231.54:/mnt/raid6-storage/storage >> 49153 0 Y 2792 >> Brick 10.0.231.55:/mnt/raid6-storage/storage >> 49153 0 Y 13590 >> Brick 10.0.231.56:/mnt/raid6-storage/storage >> 49152 0 Y 9281 >> NFS Server on localhost 2049 0 Y >> 30775 >> Quota Daemon on localhost N/A N/A Y >> 30781 >> NFS Server on 10.0.231.54 2049 0 Y >> 2817 >> Quota Daemon on 10.0.231.54 N/A N/A Y >> 2824 >> NFS Server on 10.0.231.51 2049 0 Y >> 710 >> Quota Daemon on 10.0.231.51 N/A N/A Y >> 719 >> NFS Server on 10.0.231.52 2049 0 Y >> 9090 >> Quota Daemon on 10.0.231.52 N/A N/A Y >> 9098 >> NFS Server on 10.0.231.55 2049 0 Y >> 13611 >> Quota Daemon on 10.0.231.55 N/A N/A Y >> 13619 >> NFS Server on 10.0.231.56 2049 0 Y >> 9303 >> Quota Daemon on 10.0.231.56 N/A N/A Y >> 9310 >> NFS Server on 10.0.231.53 2049 0 Y >> 26304 >> Quota Daemon on 10.0.231.53 N/A N/A Y >> 26320 >> >> Task Status of Volume storage >> ------------------------------------------------------------ >> ------------------ >> There are no active volume tasks >> >> I see lots of logs related to trashcan (failed [file exists]), set xattrs >> (failed [no such file or directory]), quota (invalid arguments) in the >> brick logs, which I enabled as a feature after the upgrade this morning. >> > > Could you let us know the time (in UTC) around which this issue was seen > such that we can look at the logs around that time and see if something > went wrong. > > >> After restarting glusterd on that host, the volume came back online. >> >> I've attached logs from that host if someone can take a look. >> >> # gluster volume info storage >> >> Volume Name: storage >> Type: Distribute >> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >> Status: Started >> Number of Bricks: 7 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >> Options Reconfigured: >> nfs.disable: no >> features.trash-max-filesize: 1GB >> features.trash: on >> features.quota-deem-statfs: on >> features.inode-quota: on >> features.quota: on >> performance.readdir-ahead: on >> >> # rpm -qa |grep glusterfs >> glusterfs-fuse-3.7.13-1.el7.x86_64 >> glusterfs-cli-3.7.13-1.el7.x86_64 >> glusterfs-3.7.13-1.el7.x86_64 >> glusterfs-server-3.7.13-1.el7.x86_64 >> glusterfs-api-3.7.13-1.el7.x86_64 >> glusterfs-libs-3.7.13-1.el7.x86_64 >> glusterfs-client-xlators-3.7.13-1.el7.x86_64 >> >> >> Thanks, >> Steve >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > > --Atin >
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users