Re: [Gluster-users] NFS-Ganesha lo traffic
Thank you very much.I just noticed even without ganesha nfs i see this kind of traffic to the lo address.and the warning message about the health status only happen when i hit 100% brick utilization, so it should be fine anyway.I'll keep digging. Thanks again. -- Respectfully Mahdi A. Mahdi > Subject: Re: [Gluster-users] NFS-Ganesha lo traffic > To: mahdi.ad...@outlook.com > CC: gluster-users@gluster.org; nfs-ganesha-de...@lists.sourceforge.net > From: skod...@redhat.com > Date: Wed, 10 Aug 2016 11:05:50 +0530 > > > > On 08/09/2016 09:06 PM, Mahdi Adnan wrote: > > Hi, > > Thank you for your reply. > > > > The traffic is related to GlusterFS; > > > > 18:31:20.419056 IP 192.168.208.134.49058 > 192.168.208.134.49153: Flags > > [.], ack 3876, win 24576, options [nop,nop,TS val 247718812 ecr > > 247718772], length 0 > > 18:31:20.419080 IP 192.168.208.134.49056 > 192.168.208.134.49154: Flags > > [.], ack 11625, win 24576, options [nop,nop,TS val 247718812 ecr > > 247718772], length 0 > > 18:31:20.419084 IP 192.168.208.134.49060 > 192.168.208.134.49152: Flags > > [.], ack 9861, win 24576, options [nop,nop,TS val 247718812 ecr > > 247718772], length 0 > > 18:31:20.419088 IP 192.168.208.134.49054 > 192.168.208.134.49155: Flags > > [.], ack 4393, win 24568, options [nop,nop,TS val 247718812 ecr > > 247718772], length 0 > > 18:31:20.420084 IP 192.168.208.134.49052 > 192.168.208.134.49156: Flags > > [.], ack 5525, win 24576, options [nop,nop,TS val 247718813 ecr > > 247718773], length 0 > > 18:31:20.420092 IP 192.168.208.134.49049 > 192.168.208.134.49158: Flags > > [.], ack 6657, win 24576, options [nop,nop,TS val 247718813 ecr > > 247718773], length 0 > > 18:31:20.421065 IP 192.168.208.134.49050 > 192.168.208.134.49157: Flags > > [.], ack 4729, win 24570, options [nop,nop,TS val 247718814 ecr > > 247718774], length 0 > > > > Looks like that is the traffic coming to the bricks local to that node > (>4915* ports are used by glusterfs brick processes). It could be from > nfs-ganesha or any other glusterfs client processes (like self-heal > daemon etc). Do you see this traffic even when there is no active I/O > from the nfs-client? If so, it could be from the self-heal daemon then. > Verify if there are any files/directories to be healed. > > > Screenshot from wireshark can be found in the attachments. > > 208.134 is the server IP address, and it's looks like it talking to > > itself via the lo interface, im wondering if this is a normal behavior > > or not. > yes. It is the expected behavior when there are clients actively > accessing the volumes. > > > and regarding the Ganesha server logs, how can i debug it to find why > > the servers not responding to the requests on time ? > > I suggest again to take tcpdump. Sometimes nfs-ganesha server (glusterfs > client) may have to communicate with all the bricks over the network > (like LOOKUP) and that may result in delay if there are lots of bricks > involved. Try capturing packets from the node where the nfs-ganesha > server is running and examine the packets between any of the NFS-client > request and its corresponding reply packet. > > I usually use below cmd to capture the packets on all the interfaces - > #tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22 > > Thanks, > Soumya > > > > > > -- > > > > Respectfully* > > **Mahdi A. Mahdi* > > > > > > > >> Subject: Re: [Gluster-users] NFS-Ganesha lo traffic > >> To: mahdi.ad...@outlook.com > >> From: skod...@redhat.com > >> CC: gluster-users@gluster.org; nfs-ganesha-de...@lists.sourceforge.net > >> Date: Tue, 9 Aug 2016 18:02:01 +0530 > >> > >> > >> > >> On 08/09/2016 03:33 PM, Mahdi Adnan wrote: > >> > Hi, > >> > > >> > Im using NFS-Ganesha to access my volume, it's working fine for now but > >> > im seeing lots of traffic on the Loopback interface, in fact it's the > >> > same amount of traffic on the bonding interface, can anyone please > >> > explain to me why is this happening ? > >> > >> Could you please capture packets on those interfaces using tcpdump and > >> examine the traffic? > >> > >> > also, i got the following error in the ganesha log file; > >> > > >> > 09/08/2016 11:35:54 : epoch 57a5da0c : gfs04 : > >> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health > >> > status is unhealthy. Not sending heartbeat > >> >
Re: [Gluster-users] NFS-Ganesha lo traffic
On 08/09/2016 09:06 PM, Mahdi Adnan wrote:
Hi,
Thank you for your reply.
The traffic is related to GlusterFS;
18:31:20.419056 IP 192.168.208.134.49058 > 192.168.208.134.49153: Flags
[.], ack 3876, win 24576, options [nop,nop,TS val 247718812 ecr
247718772], length 0
18:31:20.419080 IP 192.168.208.134.49056 > 192.168.208.134.49154: Flags
[.], ack 11625, win 24576, options [nop,nop,TS val 247718812 ecr
247718772], length 0
18:31:20.419084 IP 192.168.208.134.49060 > 192.168.208.134.49152: Flags
[.], ack 9861, win 24576, options [nop,nop,TS val 247718812 ecr
247718772], length 0
18:31:20.419088 IP 192.168.208.134.49054 > 192.168.208.134.49155: Flags
[.], ack 4393, win 24568, options [nop,nop,TS val 247718812 ecr
247718772], length 0
18:31:20.420084 IP 192.168.208.134.49052 > 192.168.208.134.49156: Flags
[.], ack 5525, win 24576, options [nop,nop,TS val 247718813 ecr
247718773], length 0
18:31:20.420092 IP 192.168.208.134.49049 > 192.168.208.134.49158: Flags
[.], ack 6657, win 24576, options [nop,nop,TS val 247718813 ecr
247718773], length 0
18:31:20.421065 IP 192.168.208.134.49050 > 192.168.208.134.49157: Flags
[.], ack 4729, win 24570, options [nop,nop,TS val 247718814 ecr
247718774], length 0
Looks like that is the traffic coming to the bricks local to that node
(>4915* ports are used by glusterfs brick processes). It could be from
nfs-ganesha or any other glusterfs client processes (like self-heal
daemon etc). Do you see this traffic even when there is no active I/O
from the nfs-client? If so, it could be from the self-heal daemon then.
Verify if there are any files/directories to be healed.
Screenshot from wireshark can be found in the attachments.
208.134 is the server IP address, and it's looks like it talking to
itself via the lo interface, im wondering if this is a normal behavior
or not.
yes. It is the expected behavior when there are clients actively
accessing the volumes.
and regarding the Ganesha server logs, how can i debug it to find why
the servers not responding to the requests on time ?
I suggest again to take tcpdump. Sometimes nfs-ganesha server (glusterfs
client) may have to communicate with all the bricks over the network
(like LOOKUP) and that may result in delay if there are lots of bricks
involved. Try capturing packets from the node where the nfs-ganesha
server is running and examine the packets between any of the NFS-client
request and its corresponding reply packet.
I usually use below cmd to capture the packets on all the interfaces -
#tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22
Thanks,
Soumya
--
Respectfully*
**Mahdi A. Mahdi*
Subject: Re: [Gluster-users] NFS-Ganesha lo traffic
To: mahdi.ad...@outlook.com
From: skod...@redhat.com
CC: gluster-users@gluster.org; nfs-ganesha-de...@lists.sourceforge.net
Date: Tue, 9 Aug 2016 18:02:01 +0530
On 08/09/2016 03:33 PM, Mahdi Adnan wrote:
> Hi,
>
> Im using NFS-Ganesha to access my volume, it's working fine for now but
> im seeing lots of traffic on the Loopback interface, in fact it's the
> same amount of traffic on the bonding interface, can anyone please
> explain to me why is this happening ?
Could you please capture packets on those interfaces using tcpdump and
examine the traffic?
> also, i got the following error in the ganesha log file;
>
> 09/08/2016 11:35:54 : epoch 57a5da0c : gfs04 :
> ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
> status is unhealthy. Not sending heartbeat
> 09/08/2016 11:46:04 : epoch 57a5da0c : gfs04 :
> ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
> status is unhealthy. Not sending heartbeat
> 09/08/2016 11:54:39 : epoch 57a5da0c : gfs04 :
> ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
> status is unhealthy. Not sending heartbeat
> 09/08/2016 12:06:04 : epoch 57a5da0c : gfs04 :
> ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
> status is unhealthy. Not sending heartbeat
>
> is it something i should care about ?
Above warnings are thrown when the outstanding rpc request queue count
doesn't change within two heartbeats, in other words the server may be
taking a while to process the requests and responding slowly to its
clients.
Thanks,
Soumya
>
> My ganesha config is the following;
>
>
> EXPORT{
> Export_Id = 1 ;
> Path = "/vlm02";
>
> FSAL {
> name = GLUSTER;
> hostname = "gfs04";
> volume = "vlm02";
> }
>
> Access_type = RW;
> Disable_ACL = TRUE;
> Squash = No_root_squash;
> Protocols = "3" ;
> Transports = "TCP";
> }
>
>
> Im accessing it via a floating ip assigned by CTDB.
>
>
> Thank you.
> --
>
> Respectfully*
> **Mahdi A. Mahdi*
>
>
>
> ___
> Gluster-u
Re: [Gluster-users] NFS-Ganesha lo traffic
On 08/09/2016 03:33 PM, Mahdi Adnan wrote:
Hi,
Im using NFS-Ganesha to access my volume, it's working fine for now but
im seeing lots of traffic on the Loopback interface, in fact it's the
same amount of traffic on the bonding interface, can anyone please
explain to me why is this happening ?
Could you please capture packets on those interfaces using tcpdump and
examine the traffic?
also, i got the following error in the ganesha log file;
09/08/2016 11:35:54 : epoch 57a5da0c : gfs04 :
ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
status is unhealthy. Not sending heartbeat
09/08/2016 11:46:04 : epoch 57a5da0c : gfs04 :
ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
status is unhealthy. Not sending heartbeat
09/08/2016 11:54:39 : epoch 57a5da0c : gfs04 :
ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
status is unhealthy. Not sending heartbeat
09/08/2016 12:06:04 : epoch 57a5da0c : gfs04 :
ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
status is unhealthy. Not sending heartbeat
is it something i should care about ?
Above warnings are thrown when the outstanding rpc request queue count
doesn't change within two heartbeats, in other words the server may be
taking a while to process the requests and responding slowly to its clients.
Thanks,
Soumya
My ganesha config is the following;
EXPORT{
Export_Id = 1 ;
Path = "/vlm02";
FSAL {
name = GLUSTER;
hostname = "gfs04";
volume = "vlm02";
}
Access_type = RW;
Disable_ACL = TRUE;
Squash = No_root_squash;
Protocols = "3" ;
Transports = "TCP";
}
Im accessing it via a floating ip assigned by CTDB.
Thank you.
--
Respectfully*
**Mahdi A. Mahdi*
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] NFS-Ganesha lo traffic
Hi,
Im using NFS-Ganesha to access my volume, it's working fine for now but im
seeing lots of traffic on the Loopback interface, in fact it's the same amount
of traffic on the bonding interface, can anyone please explain to me why is
this happening ?also, i got the following error in the ganesha log file;
09/08/2016 11:35:54 : epoch 57a5da0c : gfs04 :
ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status
is unhealthy. Not sending heartbeat09/08/2016 11:46:04 : epoch 57a5da0c :
gfs04 : ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
status is unhealthy. Not sending heartbeat09/08/2016 11:54:39 : epoch 57a5da0c
: gfs04 : ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN
:Health status is unhealthy. Not sending heartbeat09/08/2016 12:06:04 : epoch
57a5da0c : gfs04 : ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS
:WARN :Health status is unhealthy. Not sending heartbeat
is it something i should care about ?
My ganesha config is the following;
EXPORT{ Export_Id = 1 ; Path = "/vlm02";
FSAL { name = GLUSTER; hostname = "gfs04"; volume = "vlm02"; }
Access_type = RW; Disable_ACL = TRUE; Squash = No_root_squash; Protocols
= "3" ; Transports = "TCP";}
Im accessing it via a floating ip assigned by CTDB.
Thank you.
--
Respectfully
Mahdi A. Mahdi
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
