[Gluster-users] Terrible performance when a gluster node is down/rebooting

2016-08-30 Thread David Cowe 
Hello,

We have a 5 node gluster setup with the following configuration;


[root@gb0015nasslow01 Test]# gluster vol info all

Volume Name: data
Type: Tier
Volume ID: 702daa3d-b3fa-4e66-bcec-a13f7ec1d47d
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: glusternasbackup01:/bricks/hotbrick/brick
Brick2: glusternasfast02:/bricks/brick4/brick
Brick3: glusternasfast01:/bricks/brick3/brick
Cold Tier:
Cold Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick4: glusternasslow01:/bricks/brick1/brick
Brick5: glusternasslow02:/bricks/brick2/brick
Brick6: glusternasbackup01:/bricks/coldbrick/brick
Options Reconfigured:
performance.write-behind-window-size: 100mb
performance.cache-size: 20GB
storage.batch-fsync-delay-usec: 0
server.allow-insecure: on
performance.stat-prefetch: off
performance.readdir-ahead: on
features.ctr-enabled: on
cluster.tier-mode: cache


When we shutdown/reboot/pull the power from 1 of the nodes, the other
servers are very laggy when even doing an ls within the glusterfs mounted
filesystem (/glusterdata).

df -h
Filesystem   Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  4.7G   46G  10% /
devtmpfs  48G 0   48G   0% /dev
tmpfs 48G 0   48G   0% /dev/shm
tmpfs 48G   66M   48G   1% /run
tmpfs 48G 0   48G   0% /sys/fs/cgroup
/dev/sdb1110T  221M  110T   1% /bricks/brick1
/dev/sda2497M  201M  297M  41% /boot
/dev/sda1200M  9.5M  191M   5% /boot/efi
/dev/mapper/centos-home  225G   33M  224G   1% /home
localhost:data   109T  464G  108T   1% /glusterdata
tmpfs9.5G 0  9.5G   0% /run/user/0

During the time the node is down or coming back up, we not only get the lag
described above on the server but file reads and writes to the gluster
volume pretty much stall.




Can anyone give me some much needed advice as surely this should not be the
case?




*packages installed;*

yum list gluster*

Installed Packages
glusterfs.x86_643.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-api.x86_643.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-cli.x86_643.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-client-xlators.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-fuse.x86_64   3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-libs.x86_64   3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-server.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64


yum list samba*

Installed Packages
samba.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client-libs.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common.noarch
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-libs.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-tools.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-libs.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-vfs-glusterfs.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-clients.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-modules.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64


yum list ctdb*

Installed Packages
ctdb.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64


Background - We use CTDB for VIP's and heartbeat and we present our gluster
volume via samba to our Windows clients.

Regards,
David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Can't delete files from gluster volume in Windows

2016-08-02 Thread David Cowe 
Hello,

Glusterfs 3.7.10

We have a glusterfs volume made up of 1 brick on each of our 4 nodes.

The volume is setup using tiering. The hot tier has 2 bricks in a replicate
and the cold tier has 2 bricks in a replicate.

 We use samba (4.2) and ctdb to mount the volume to our windows clients via
cifs.

We cannot delete a file from the cifs mounted volume on Windows. The file
deletes ok on the Windows side without error but it does not delete from
the glusterfs volume on the storage nodes! When refreshing the Windows
cifs mounted volume (using f5), the file reappears.

 We can install the gluster client on a Linux machine and mount the gluster
volume and delete a file without any of the above issues of the file
reappearing.

We can also do this on Linux mounting it via nfs.


Our problem is to do with Gluster and Samba. Any thoughts?


Regards,

David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] (no subject)

2016-08-02 Thread David Cowe 
Hello,

Glusterfs 3.7.10

We have a glusterfs volume made up of 1 brick on each of our 4 nodes.

The volume is setup using tiering. The hot tier has 2 bricks in a replicate
and the cold tier has 2 bricks in a replicate.

We use samba (4.2) and ctdb to mount the volume to our windows clients via
cifs.

We cannot delete a file from the cifs mounted volume on Windows. The file
deletes ok on the Windows side without error but it does not delete from
the glusterfs volume on the storage nodes! When refreshing the Windows
cifs mounted volume (using f5), the file reappears.

We can install the gluster client on a Linux machine and mount the gluster
volume and delete a file without any of the above issues of the file
reappearing. We can also do this on Linux mounting it via nfs.

Our problem is to do with Gluster and Samba.

Any thoughts?

Regards,
David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users