On Mon, Sep 3, 2018 at 11:17 AM Karthik Subrahmanya <ksubr...@redhat.com> wrote:
> Hey, > > We need some more information to debug this. > I think you missed to send the output of 'gluster volume info <volname>'. > Can you also provide the bricks, shd and glfsheal logs as well? > In the setup how many peers are present? You also mentioned that "one of > the file servers have two processes for each of the volumes instead of one > per volume", which process are you talking about here? > Also provide the "ps aux | grep gluster" output. > > Regards, > Karthik > > On Sat, Sep 1, 2018 at 12:10 AM Johnson, Tim <t...@uic.edu> wrote: > >> Thanks for the reply. >> >> >> >> I have attached the gluster.log file from the host that it is >> happening to at this time. >> >> It does change which host it does this on. >> >> >> >> Thanks. >> >> >> >> *From: *Atin Mukherjee <amukh...@redhat.com> >> *Date: *Friday, August 31, 2018 at 1:03 PM >> *To: *"Johnson, Tim" <t...@uic.edu> >> *Cc: *Karthik Subrahmanya <ksubr...@redhat.com>, Ravishankar N < >> ravishan...@redhat.com>, "gluster-users@gluster.org" < >> gluster-users@gluster.org> >> *Subject: *Re: [Gluster-users] Transport endpoint is not connected : >> issue >> >> >> >> Can you please pass all the gluster log files from the server where the >> transport end point not connected error is reported? As restarting glusterd >> didn’t solve this issue, I believe this isn’t a stale port problem but >> something else. Also please provide the output of ‘gluster v info <volname>’ >> >> >> >> (@cc Ravi, Karthik) >> >> >> >> On Fri, 31 Aug 2018 at 23:24, Johnson, Tim <t...@uic.edu> wrote: >> >> Hello all, >> >> >> >> We have a gluster replicate (with arbiter) volumes that we are >> getting “Transport endpoint is not connected” with on a rotating basis >> from each of the two file servers, and a third host that has the arbiter >> bricks on. >> >> This is happening when trying to run a heal on all the volumes on the >> gluster hosts When I get the status of all the volumes all looks good. >> >> This behavior seems to be a forshadowing of the gluster volumes >> becoming unresponsive to our vm cluster. As well as one of the file >> servers have two processes for each of the volumes instead of one per >> volume. Eventually the affected file server >> >> will drop off the listed peers. Restarting glusterd/glusterfsd on the >> affected file server does not take care of the issue, we have to bring down >> both file >> >> Servers due to the volumes not being seen by the vm cluster after the >> errors start occurring. I had seen that there were bug reports about the >> “Transport endpoint is not connected” on earlier versions of Gluster >> however had thought that >> >> It had been addressed. >> >> Dmesg did have some entries for “a possible syn flood on port *” >> which we changed the sysctl to “net.ipv4.tcp_max_syn_backlog = 2048” which >> seemed to help the syn flood messages but not the underlying volume issues. >> >> I have put the versions of all the Gluster packages installed below >> as well as the “Heal” and “Status” commands showing the volumes are >> >> >> >> This has just started happening but cannot definitively say if >> this started occurring after an update or not. >> >> >> >> >> >> Thanks for any assistance. >> >> >> >> >> >> Running Heal : >> >> >> >> gluster volume heal ovirt_engine info >> >> Brick ****1.rrc.local:/bricks/brick0/ovirt_engine >> >> Status: Connected >> >> Number of entries: 0 >> >> >> >> Brick ****3.rrc.local:/bricks/brick0/ovirt_engine >> >> Status: Transport endpoint is not connected >> >> Number of entries: - >> >> >> >> Brick *****3.rrc.local:/bricks/arb-brick/ovirt_engine >> >> Status: Transport endpoint is not connected >> >> Number of entries: - >> >> >> >> >> >> Running status : >> >> >> >> gluster volume status ovirt_engine >> >> Status of volume: ovirt_engine >> >> Gluster process TCP Port RDMA Port Online >> Pid >> >> >> ------------------------------------------------------------------------------ >> >> Brick*****.rrc.local:/bricks/brick0/ov >> >> irt_engine 49152 0 Y >> 5521 >> >> Brick fs2-tier3.rrc.local:/bricks/brick0/ov >> >> irt_engine 49152 0 Y >> 6245 >> >> Brick ****.rrc.local:/bricks/arb-b >> >> rick/ovirt_engine 49152 0 Y >> 3526 >> >> Self-heal Daemon on localhost N/A N/A Y >> 5509 >> >> Self-heal Daemon on ***.rrc.local N/A N/A Y 6218 >> >> Self-heal Daemon on ***.rrc.local N/A N/A Y 3501 >> >> Self-heal Daemon on ****.rrc.local N/A N/A Y 3657 >> >> Self-heal Daemon on *****.rrc.local N/A N/A Y 3753 >> >> Self-heal Daemon on ****.rrc.local N/A N/A Y 17284 >> >> >> >> Task Status of Volume ovirt_engine >> >> >> ------------------------------------------------------------------------------ >> >> There are no active volume tasks >> >> >> >> >> >> >> >> >> >> /etc/glusterd.vol. : >> >> >> >> >> >> volume management >> >> type mgmt/glusterd >> >> option working-directory /var/lib/glusterd >> >> option transport-type socket,rdma >> >> option transport.socket.keepalive-time 10 >> >> option transport.socket.keepalive-interval 2 >> >> option transport.socket.read-fail-log off >> >> option ping-timeout 0 >> >> option event-threads 1 >> >> option rpc-auth-allow-insecure on >> >> # option transport.address-family inet6 >> >> # option base-port 49152 >> >> end-volume >> >> >> >> >> >> >> >> >> >> >> >> rpm -qa |grep gluster >> >> glusterfs-3.12.13-1.el7.x86_64 >> >> glusterfs-gnfs-3.12.13-1.el7.x86_64 >> >> glusterfs-api-3.12.13-1.el7.x86_64 >> >> glusterfs-cli-3.12.13-1.el7.x86_64 >> >> glusterfs-client-xlators-3.12.13-1.el7.x86_64 >> >> glusterfs-fuse-3.12.13-1.el7.x86_64 >> >> centos-release-gluster312-1.0-2.el7.centos.noarch >> >> glusterfs-rdma-3.12.13-1.el7.x86_64 >> >> glusterfs-libs-3.12.13-1.el7.x86_64 >> >> glusterfs-server-3.12.13-1.el7.x86_64 >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> -- >> >> - Atin (atinm) >> >
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users