Re: [Gluster-users] possible kernel panic with glusterd
Could the oom killer be kicking in? is this a repeatable issue? -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] possible kernel panic with glusterd
That file exists but is empty. Joe On Wed, Jan 25, 2017 at 7:44 AM Samikshan Bairagya wrote: > > > On 01/25/2017 06:57 PM, Joseph Lorenzini wrote: > > Hi Atin, > > > > I assume you are referring to the /var/log/glustershd.log. If so, that > file > > never gets created. > > You'd find the glusterd log file as > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log AFAR for glusterfs-3.8. > > ~ Samikshan > > > > > Joe > > > > On Wed, Jan 25, 2017 at 6:14 AM Atin Mukherjee > wrote: > > > >> On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini > >> wrote: > >> > >> Hi all, > >> > >> I have recently started exploring the DFS solution space and was doing > >> some basic setup and testing with gluster. I set up a pool of three > nodes > >> following the quick start guide. That seemed to work fine. > >> > >> However, shortly after that, I noticed that one of the servers in the > pool > >> was becoming non-responsive -- as in the entire VM was completely hung > and > >> i had to use the hypervisor to force a reboot. I sshed into the server > and > >> startd poking around. glusterd was shut off. I started it up and the > >> following happened: > >> > >> Message from syslogd at Jan 25 05:20:47 ... > >> kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for > >> 22s! [glusterd:2374] > >> > >> > >> Could you attach the glusterd log file to enable us to look at why > >> glusterd got shutdown? > >> > >> > >> > >> At which point, the VM became completely unresponsive again. > >> > >> All servers are the same. They are running centos 7.3, linux kernel > >> 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. > >> > >> Since I just started investigating gluster, it is certainly possible > that > >> I misconfigured something on that one node. However, a kernel hang/panic > >> seems like an excessive response :). If anyone would have any ideas or > >> suggestions about what may be happening here or additional places I > should > >> look into to find out what is going on, I am all ears. > >> > >> Thanks, > >> Joe > >> > >> ___ > >> Gluster-users mailing list > >> Gluster-users@gluster.org > >> http://lists.gluster.org/mailman/listinfo/gluster-users > >> > >> > >> > >> > >> -- > >> > >> ~ Atin (atinm) > >> > > > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] possible kernel panic with glusterd
On 01/25/2017 06:57 PM, Joseph Lorenzini wrote: Hi Atin, I assume you are referring to the /var/log/glustershd.log. If so, that file never gets created. You'd find the glusterd log file as /var/log/glusterfs/etc-glusterfs-glusterd.vol.log AFAR for glusterfs-3.8. ~ Samikshan Joe On Wed, Jan 25, 2017 at 6:14 AM Atin Mukherjee wrote: On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini wrote: Hi all, I have recently started exploring the DFS solution space and was doing some basic setup and testing with gluster. I set up a pool of three nodes following the quick start guide. That seemed to work fine. However, shortly after that, I noticed that one of the servers in the pool was becoming non-responsive -- as in the entire VM was completely hung and i had to use the hypervisor to force a reboot. I sshed into the server and startd poking around. glusterd was shut off. I started it up and the following happened: Message from syslogd at Jan 25 05:20:47 ... kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [glusterd:2374] Could you attach the glusterd log file to enable us to look at why glusterd got shutdown? At which point, the VM became completely unresponsive again. All servers are the same. They are running centos 7.3, linux kernel 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. Since I just started investigating gluster, it is certainly possible that I misconfigured something on that one node. However, a kernel hang/panic seems like an excessive response :). If anyone would have any ideas or suggestions about what may be happening here or additional places I should look into to find out what is going on, I am all ears. Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users -- ~ Atin (atinm) ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] possible kernel panic with glusterd
Hi Atin, I assume you are referring to the /var/log/glustershd.log. If so, that file never gets created. Joe On Wed, Jan 25, 2017 at 6:14 AM Atin Mukherjee wrote: > On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini > wrote: > > Hi all, > > I have recently started exploring the DFS solution space and was doing > some basic setup and testing with gluster. I set up a pool of three nodes > following the quick start guide. That seemed to work fine. > > However, shortly after that, I noticed that one of the servers in the pool > was becoming non-responsive -- as in the entire VM was completely hung and > i had to use the hypervisor to force a reboot. I sshed into the server and > startd poking around. glusterd was shut off. I started it up and the > following happened: > > Message from syslogd at Jan 25 05:20:47 ... > kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for > 22s! [glusterd:2374] > > > Could you attach the glusterd log file to enable us to look at why > glusterd got shutdown? > > > > At which point, the VM became completely unresponsive again. > > All servers are the same. They are running centos 7.3, linux kernel > 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. > > Since I just started investigating gluster, it is certainly possible that > I misconfigured something on that one node. However, a kernel hang/panic > seems like an excessive response :). If anyone would have any ideas or > suggestions about what may be happening here or additional places I should > look into to find out what is going on, I am all ears. > > Thanks, > Joe > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > > -- > > ~ Atin (atinm) > ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] possible kernel panic with glusterd
On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini wrote: > Hi all, > > I have recently started exploring the DFS solution space and was doing > some basic setup and testing with gluster. I set up a pool of three nodes > following the quick start guide. That seemed to work fine. > > However, shortly after that, I noticed that one of the servers in the pool > was becoming non-responsive -- as in the entire VM was completely hung and > i had to use the hypervisor to force a reboot. I sshed into the server and > started poking around. glusterd was shut off. I started it up and the > following happened: > > Message from syslogd at Jan 25 05:20:47 ... > kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for > 22s! [glusterd:2374] > > Could you attach the glusterd log file to enable us to look at why glusterd got shutdown? > > At which point, the VM became completely unresponsive again. > > All servers are the same. They are running centos 7.3, linux kernel > 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. > > Since I just started investigating gluster, it is certainly possible that > I misconfigured something on that one node. However, a kernel hang/panic > seems like an excessive response :). If anyone would have any ideas or > suggestions about what may be happening here or additional places I should > look into to find out what is going on, I am all ears. > > Thanks, > Joe > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > -- ~ Atin (atinm) ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] possible kernel panic with glusterd
Hi all, I have recently started exploring the DFS solution space and was doing some basic setup and testing with gluster. I set up a pool of three nodes following the quick start guide. That seemed to work fine. However, shortly after that, I noticed that one of the servers in the pool was becoming non-responsive -- as in the entire VM was completely hung and i had to use the hypervisor to force a reboot. I sshed into the server and started poking around. glusterd was shut off. I started it up and the following happened: Message from syslogd at Jan 25 05:20:47 ... kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [glusterd:2374] At which point, the VM became completely unresponsive again. All servers are the same. They are running centos 7.3, linux kernel 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. Since I just started investigating gluster, it is certainly possible that I misconfigured something on that one node. However, a kernel hang/panic seems like an excessive response :). If anyone would have any ideas or suggestions about what may be happening here or additional places I should look into to find out what is going on, I am all ears. Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
