Re: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Ben Turner]

- Original Message -
> From: "Ben Turner" <btur...@redhat.com>
> To: "Julio Guevara" <julioguevara...@gmail.com>
> Cc: gluster-users@gluster.org
> Sent: Thursday, June 15, 2017 6:10:58 PM
> Subject: Re: [Gluster-users] Transport Endpoint Not connected while running 
> sysbench on Gluster Volume
> 
> 
> 
> - Original Message -
> > From: "Julio Guevara" <julioguevara...@gmail.com>
> > To: "Ben Turner" <btur...@redhat.com>
> > Sent: Thursday, June 15, 2017 5:52:26 PM
> > Subject: Re: [Gluster-users] Transport Endpoint Not connected while running
> > sysbench on Gluster Volume
> > 
> > I stumble upon the problem.
> > 
> > We are using deep security agent (da_agent) as our main antivirus. When the
> > antivirus gets activated it installs kernel modules:
> >   redirfs
> >   gsch
> > 
> > Apparently when this modules are present and loaded to the kernel, I see
> > all the issues that i have described here.
> > Once I uninstall the agent and reboot the system (To make sure modules are
> > unloaded) glusterfs works without any issue.
> > This is the sofware version that i'm using if it is useful for anybody:
> > 
> >   CentOS 6.8
> >   kernel2.6.32-696.3.1.el6
> >   ds_agent   9.6.2-7723.el6 tried with ds_agent 9.6.2-7888.el6
> >  same issue.
> >   glusterfs-server  3.8.12-1.el6
> > 
> > @Ben the tail I sent before includes both server and client logs, even
> > bricks.
> 
> Hmm, maybe the security SW is killing / interfering some how with the gluster
> stack?  Do you know the expected behavior of the antivirus when is sees
> binaries and / or behavior it doesn't recognize?  Maybe FUSE being in user
> space is tripping it up?  Is there any way to configure the anitvirus to
> white list / not interfere with the components of the gluster stack?

I just did a quick google and saw:

http://docs.trendmicro.com/all/ent/ds/v9.5_sp1/en-us/DS_Agent-Linux_9.5_SP1_readme.txt

   - Anti-Malware is unable to scan fuse-based file-system if the 
 mount owner is not root, and the mount does not allow other users to 
 access. [26265]

So it would appear that there have been some issues with FUSE based file 
systems.  It may be worth reaching out to the vendor if you have support and 
see if there are any known issues with FUSE based systems.  In the meantime you 
may want to try NFS if you NEED the antivirus else you could leave it disabled 
until you get the issue sorted.

-b


> 
> -b
> 
> 
> > 
> > Thanks
> > Julio Guevara
> > 
> > On Wed, Jun 14, 2017 at 11:11 PM, Ben Turner <btur...@redhat.com> wrote:
> > 
> > > - Original Message -
> > > > From: "Julio Guevara" <julioguevara...@gmail.com>
> > > > To: gluster-users@gluster.org
> > > > Sent: Tuesday, June 13, 2017 4:43:06 PM
> > > > Subject: [Gluster-users] Transport Endpoint Not connected while running
> > >  sysbench on Gluster Volume
> > > >
> > > > I'm having a hard time trying to get a gluster volume up and running. I
> > > have
> > > > setup other gluster volumes on other systems without much problems but
> > > this
> > > > one is killing me.
> > > >
> > > > The gluster vol was created with the command:
> > > > gluster volume create mariadb_gluster_volume
> > > > laeft-dccdb01p:/export/mariadb/brick
> > > >
> > > > I had to lower frame-timeout since the system would become unresponsive
> > > until
> > > > the frame failed by timeout:
> > > > gluster volume set mariadb_gluster_volume networking.frame-timeout 5
> > > >
> > > > running gluster version: glusterfs 3.8.12
> > > >
> > > > The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> > > > --file-num=64 prepare
> > > >
> > > > sysbench version: sysbench 0.4.12-5.el6
> > > >
> > > > kernel version: 2.6.32-696.1.1.el6
> > > >
> > > > centos: 6.8
> > > >
> > > > Issue: Whenever I run the sysbench over the mount
> > > > /var/lib/mysql_backups
> > > I
> > > > get the error that is shown on the log output.
> > > >
> > > > It is a constant issue, I can reproduce it when I start increasing the
> > > > --file-num for sysbench above 3.
> > >
> > > It looks like you may be seeing a crash.  If you look at
> > > /var/log/messages
> >

Re: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Ben Turner]

- Original Message -
> From: "Julio Guevara" <julioguevara...@gmail.com>
> To: "Ben Turner" <btur...@redhat.com>
> Sent: Thursday, June 15, 2017 5:52:26 PM
> Subject: Re: [Gluster-users] Transport Endpoint Not connected while running 
> sysbench on Gluster Volume
> 
> I stumble upon the problem.
> 
> We are using deep security agent (da_agent) as our main antivirus. When the
> antivirus gets activated it installs kernel modules:
>   redirfs
>   gsch
> 
> Apparently when this modules are present and loaded to the kernel, I see
> all the issues that i have described here.
> Once I uninstall the agent and reboot the system (To make sure modules are
> unloaded) glusterfs works without any issue.
> This is the sofware version that i'm using if it is useful for anybody:
> 
>   CentOS 6.8
>   kernel2.6.32-696.3.1.el6
>   ds_agent   9.6.2-7723.el6 tried with ds_agent 9.6.2-7888.el6
>  same issue.
>   glusterfs-server  3.8.12-1.el6
> 
> @Ben the tail I sent before includes both server and client logs, even
> bricks.

Hmm, maybe the security SW is killing / interfering some how with the gluster 
stack?  Do you know the expected behavior of the antivirus when is sees 
binaries and / or behavior it doesn't recognize?  Maybe FUSE being in user 
space is tripping it up?  Is there any way to configure the anitvirus to white 
list / not interfere with the components of the gluster stack?

-b


> 
> Thanks
> Julio Guevara
> 
> On Wed, Jun 14, 2017 at 11:11 PM, Ben Turner <btur...@redhat.com> wrote:
> 
> > - Original Message -
> > > From: "Julio Guevara" <julioguevara...@gmail.com>
> > > To: gluster-users@gluster.org
> > > Sent: Tuesday, June 13, 2017 4:43:06 PM
> > > Subject: [Gluster-users] Transport Endpoint Not connected while running
> >  sysbench on Gluster Volume
> > >
> > > I'm having a hard time trying to get a gluster volume up and running. I
> > have
> > > setup other gluster volumes on other systems without much problems but
> > this
> > > one is killing me.
> > >
> > > The gluster vol was created with the command:
> > > gluster volume create mariadb_gluster_volume
> > > laeft-dccdb01p:/export/mariadb/brick
> > >
> > > I had to lower frame-timeout since the system would become unresponsive
> > until
> > > the frame failed by timeout:
> > > gluster volume set mariadb_gluster_volume networking.frame-timeout 5
> > >
> > > running gluster version: glusterfs 3.8.12
> > >
> > > The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> > > --file-num=64 prepare
> > >
> > > sysbench version: sysbench 0.4.12-5.el6
> > >
> > > kernel version: 2.6.32-696.1.1.el6
> > >
> > > centos: 6.8
> > >
> > > Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups
> > I
> > > get the error that is shown on the log output.
> > >
> > > It is a constant issue, I can reproduce it when I start increasing the
> > > --file-num for sysbench above 3.
> >
> > It looks like you may be seeing a crash.  If you look at /var/log/messages
> > on all of the clients / servers do you see any crashes / seg faults / ABRT
> > messages in the log?  If so can you open a BZ with the core / other info
> > here?  Here is an example of a crash on one of the bricks:
> >
> > http://lists.gluster.org/pipermail/gluster-users/2016-February/025460.html
> >
> > My guess is something is happening client sidesince we don't see anything
> > in the server logs, check the client mount
> > log(/var/log/glusterfs/.log
> > and the messages file on your client.  Also check messages on the servers.
> > If you see anything shoot us out the info and lets get a BZ open, if not
> > maybe someone else on the list has some other ideas.
> >
> > -b
> >
> > >
> > >
> > >
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Ben Turner]

- Original Message -
> From: "Julio Guevara" <julioguevara...@gmail.com>
> To: gluster-users@gluster.org
> Sent: Tuesday, June 13, 2017 4:43:06 PM
> Subject: [Gluster-users] Transport Endpoint Not connected while running   
> sysbench on Gluster Volume
> 
> I'm having a hard time trying to get a gluster volume up and running. I have
> setup other gluster volumes on other systems without much problems but this
> one is killing me.
> 
> The gluster vol was created with the command:
> gluster volume create mariadb_gluster_volume
> laeft-dccdb01p:/export/mariadb/brick
> 
> I had to lower frame-timeout since the system would become unresponsive until
> the frame failed by timeout:
> gluster volume set mariadb_gluster_volume networking.frame-timeout 5
> 
> running gluster version: glusterfs 3.8.12
> 
> The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> --file-num=64 prepare
> 
> sysbench version: sysbench 0.4.12-5.el6
> 
> kernel version: 2.6.32-696.1.1.el6
> 
> centos: 6.8
> 
> Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups I
> get the error that is shown on the log output.
> 
> It is a constant issue, I can reproduce it when I start increasing the
> --file-num for sysbench above 3.

It looks like you may be seeing a crash.  If you look at /var/log/messages on 
all of the clients / servers do you see any crashes / seg faults / ABRT 
messages in the log?  If so can you open a BZ with the core / other info here?  
Here is an example of a crash on one of the bricks:

http://lists.gluster.org/pipermail/gluster-users/2016-February/025460.html

My guess is something is happening client sidesince we don't see anything in 
the server logs, check the client mount 
log(/var/log/glusterfs/.log and the messages file on your client.  
Also check messages on the servers.  If you see anything shoot us out the info 
and lets get a BZ open, if not maybe someone else on the list has some other 
ideas.

-b

> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Julio Guevara]

I'm having a hard time trying to get a gluster volume up and running. I
have setup other gluster volumes on other systems without much problems but
this one is killing me.

The gluster vol was created with the command:
gluster volume create mariadb_gluster_volume
laeft-dccdb01p:/export/mariadb/brick

I had to lower frame-timeout since the system would become unresponsive
until the frame failed by timeout:
gluster volume set mariadb_gluster_volume networking.frame-timeout 5

running gluster version: glusterfs 3.8.12

The workload i'm using is: sysbench --test=fileio --file-total-size=4G
--file-num=64 prepare

sysbench version: sysbench 0.4.12-5.el6

kernel version: 2.6.32-696.1.1.el6

centos: 6.8

Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups I
get the error that is shown on the log output.

It is a constant issue, I can reproduce it when I start increasing the
--file-num for sysbench above 3.


all.logs
Description: Binary data
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Julio Guevara]

Also, this is the profile output of this Volume:

gluster> volume profile mariadb_gluster_volume info cumulative
Brick: laeft-dccdb01p.core.epay.us.loc:/export/mariadb_backup/brick
---
Cumulative Stats:
   Block Size:  16384b+   32768b+
65536b+
 No. of Reads:0 0
  0
No. of Writes:83391465750
 102911

   Block Size: 131072b+
 No. of Reads:   33
No. of Writes: 8551
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 -   ---   ---   ---   
 
  0.00   0.00 us   0.00 us   0.00 us 26
 FORGET
  0.00   0.00 us   0.00 us   0.00 us 33
RELEASE
  0.00   0.00 us   0.00 us   0.00 us 92
 RELEASEDIR
  0.00 102.25 us  70.00 us 155.00 us  4
 SETXATTR
  0.00 159.80 us  97.00 us 207.00 us  5
RMDIR
  0.00 266.75 us 121.00 us 675.00 us  4
SETATTR
  0.00 100.62 us  29.00 us 700.00 us 16
INODELK
  0.00  84.33 us  29.00 us 477.00 us 33
FLUSH
  0.00  68.16 us  34.00 us 165.00 us 92
OPENDIR
  0.01  88.35 us  21.00 us 608.00 us 92
 STAT
  0.01 754.04 us 262.00 us   10104.00 us 25
 CREATE
  0.02 169.04 us  27.00 us 997.00 us179
 READDIRP
  0.04 150.90 us  43.00 us1867.00 us365
 LOOKUP
  0.04   16330.75 us 297.00 us   46360.00 us  4
MKDIR
  0.187896.70 us  71.00 us  256814.00 us 33
 READ
  0.68 466.93 us  19.00 us1848.00 us   2119
 STATFS
  2.39  151339.17 us 227.00 us  540998.00 us 23
 UNLINK
  5.49  320155.28 us2035.00 us 1273394.00 us 25
FSYNC
 31.28  69.00 us  30.00 us   11447.00 us 660603
WRITE
 59.84 10899586.88 us 5827342.00 us 13921169.00 us  8
 OPEN

Duration: 71425 seconds
   Data Read: 4325376 bytes
Data Written: 29195534336 bytes


As you can see OPEN Fop take the most amount of time and they normally
timeout, even with default value for networking.frame-timeout

Thanks

On Tue, Jun 13, 2017 at 3:43 PM, Julio Guevara 
wrote:

> I'm having a hard time trying to get a gluster volume up and running. I
> have setup other gluster volumes on other systems without much problems but
> this one is killing me.
>
> The gluster vol was created with the command:
> gluster volume create mariadb_gluster_volume laeft-dccdb01p:/export/
> mariadb/brick
>
> I had to lower frame-timeout since the system would become unresponsive
> until the frame failed by timeout:
> gluster volume set mariadb_gluster_volume networking.frame-timeout 5
>
> running gluster version: glusterfs 3.8.12
>
> The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> --file-num=64 prepare
>
> sysbench version: sysbench 0.4.12-5.el6
>
> kernel version: 2.6.32-696.1.1.el6
>
> centos: 6.8
>
> Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups I
> get the error that is shown on the log output.
>
> It is a constant issue, I can reproduce it when I start increasing the
> --file-num for sysbench above 3.
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users