Re: [Gluster-users] Split-brain after uploading file
any thoughts? Miloš > 30. 11. 2015 v 23:45, Miloš Kozák : > > I am using Gluster for a few years without any significant issue (after I > tweaked configuration for v3.5). My configuration is as follows: > > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > performance.io-thread-count: 6 > network.ping-timeout: 2 > performance.cache-max-file-size: 0 > performance.flush-behind: on > features.barrier: disable > snap-max-soft-limit: 7 > auto-delete: on > > I use it for running virtual servers on top such a volume. Currently I run > this version of Gluster: > > glusterfs-cli-3.6.5-1.el6.x86_64 > glusterfs-3.6.5-1.el6.x86_64 > glusterfs-api-3.6.5-1.el6.x86_64 > glusterfs-server-3.6.5-1.el6.x86_64 > glusterfs-libs-3.6.5-1.el6.x86_64 > glusterfs-fuse-3.6.5-1.el6.x86_64 > > With recent CentOS 6. > > I have experienced an issue when I move some files from an hdd onto gluster > volume such that one node gets overloaded in the middle of file upload. > Therefore, I decided to upload it through ssh onto other server than where > original images are store. I know that this sounds just weird, but it does > not lead to overloading! > > Along these lines, I decided to upload 10G image onto gluster volume and the > upload speed varied, but no overloading at all… Right after upload was done I > realized that some virtuals are not running properly. Hence I checked heal > status where I discoverd that 4 images are in split-brain state. I had to act > quickly, so I resolved the split brain, and let gluster heal. When heal was > done everything works… > > However, I have got a few more VMs to upload, and I am not sure what can > happen.. > > My volume configuration: > > Volume Name: ph-fs-0 > Type: Replicate > Volume ID: 71ac6456-03e4-4bb3-a624-937f4605b2cb > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: 10.11.100.1:/gfs/s3-sata-10k/fs > Brick2: 10.11.100.2:/gfs/s3-sata-10k/fs > Options Reconfigured: > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > performance.io-thread-count: 6 > network.ping-timeout: 2 > performance.cache-max-file-size: 0 > performance.flush-behind: on > features.barrier: disable > snap-max-soft-limit: 7 > auto-delete: on > > > and logs are attached. > > Miloš > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Configuring Ganesha and gluster on separate nodes?
Hi Soumya, Kaleb, all:
Thanks for the response!
Quick follow-up to this question - We tried running ganesha and gluster on
two separate machines and the configuration seems to be working without
issues.
Follow-up question I have is this: what changes do I need to make to put
the Ganesha in active active HA mode - where backend gluster and ganesha
will be on a different node. I am using the instructions here for putting
Ganesha in HA mode. http://www.slideshare.net/SoumyaKoduri/high-49117846.
This presentation refers to commands like gluster
cluster.enable-shared-storage to enable HA.
1. Here is the config I am hoping to achieve:
glusterA and glusterB on individual bare metals - both in Trusted pool,
with volume gvol0 up and running.
Ganesha 1 and 2 on machines ganesha1, and ganesha1. And my gluster storage
will be on a third machine gluster1. (with a peer on another machine
gluster2).
Ganesha node1: on a VM ganeshaA.
Ganesha node2: on another vm GaneshaB.
I would like to know what it takes to put ganeshaA and GaneshaB in Active
Active HA mode. Is it technically possible?
a. How do commands like cluster.enable-shared-storage work in this case?
b. where does this command need to be run? on the ganesha node, or on the
gluster nodes?
2. Also, is it possible to have multiple ganesha servers point to the same
gluster volume in the back end? say, in the configuration #1, I have
another ganesha server GaneshaC that is not clustered with ganeshaA or
ganeshaB. Can it export the volume gvol0 that ganeshaA and ganeshaB are
also exporting?
thank you!
Surya.
Regards,
Surya Ghatty
"This too shall pass"
Surya Ghatty | Software Engineer | IBM Cloud Infrastructure Services
Development | tel: (507) 316-0559 | gha...@us.ibm.com
From: Soumya Koduri
To: Surya K Ghatty/Rochester/IBM@IBMUS, gluster-users@gluster.org
Date: 11/18/2015 05:08 AM
Subject:Re: [Gluster-users] Configuring Ganesha and gluster on separate
nodes?
On 11/17/2015 10:21 PM, Surya K Ghatty wrote:
> Hi:
>
> I am trying to understand if it is technically feasible to have gluster
> nodes on one machine, and export a volume from one of these nodes using
> a nfs-ganesha server installed on a totally different machine? I tried
> the below and showmount -e does not show my volume exported. Any
> suggestions will be appreciated.
>
> 1. Here is my configuration:
>
> Gluster nodes: glusterA and glusterB on individual bare metals - both in
> Trusted pool, with volume gvol0 up and running.
> Ganesha node: on bare metal ganeshaA.
>
> 2. my ganesha.conf looks like this with IP address of glusterA in the
FSAL.
>
> FSAL {
> Name = GLUSTER;
>
> # IP of one of the nodes in the trusted pool
> *hostname = "WW.ZZ.XX.YY" --> IP address of GlusterA.*
>
> # Volume name. Eg: "test_volume"
> volume = "gvol0";
> }
>
> 3. I disabled nfs on gvol0. As you can see, *nfs.disable is set to on.*
>
> [root@glusterA ~]# gluster vol info
>
> Volume Name: gvol0
> Type: Distribute
> Volume ID: 16015bcc-1d17-4ef1-bb8b-01b7fdf6efa0
> Status: Started
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: glusterA:/data/brick0/gvol0
> Options Reconfigured:
> *nfs.disable: on*
> nfs.export-volumes: off
> features.quota-deem-statfs: on
> features.inode-quota: on
> features.quota: on
> performance.readdir-ahead: on
>
> 4. I then ran ganesha.nfsd -f /etc/ganesha/ganesha.conf -L
> /var/log/ganesha.log -N NIV_FULL_DEBUG
> Ganesha server was put in grace, no errors.
>
> 17/11/2015 10:44:40 : epoch 564b5964 : ganeshaA:
> nfs-ganesha-26426[reaper] fridgethr_freeze :RW LOCK :F_DBG :Released
> mutex 0x7f21a92818d0 (&fr->mtx) at
> /builddir/build/BUILD/nfs-ganesha-2.2.0/src/support/fridgethr.c:484
> 17/11/2015 10:44:40 : epoch 564b5964 : ganeshaA:
> nfs-ganesha-26426[reaper] nfs_in_grace :RW LOCK :F_DBG :Acquired mutex
> 0x7f21ad1f18e0 (&grace.g_mutex) at
> /builddir/build/BUILD/nfs-ganesha-2.2.0/src/SAL/nfs4_recovery.c:129
> *17/11/2015 10:44:40 : epoch 564b5964 : ganeshaA :
> nfs-ganesha-26426[reaper] nfs_in_grace :STATE :DEBUG :NFS Server IN
GRACE*
> 17/11/2015 10:44:40 : epoch 564b5964 : ganeshaA :
> nfs-ganesha-26426[reaper] nfs_in_grace :RW LOCK :F_DBG :Released mutex
> 0x7f21ad1f18e0 (&grace.g_mutex) at
> /builddir/build/BUILD/nfs-ganesha-2.2.0/src/SAL/nfs4_recovery.c:141
>
You shall still need gluster-client bits on the machine where
nfs-ganesha server is installed to export a gluster volume. Check if you
have got libgfapi.so installed on that machine.
Also, ganesha server does log the warnings if its unable to process the
EXPORT/FSAL block. Please recheck the logs if you have got any.
Thanks,
Soumya
> 5. [root@ganeshaA glusterfs]# showmount -e
> Export list for ganeshaA:
>
>
> Any suggestions on what I am missing?
>
> Regards,
>
> Surya Ghatty
>
> "This too shall pass"
>
__
Re: [Gluster-users] samba-vfs-glusterfs share problems
Just about to roll this into general use, and realized clients can't always follow symlinks. I've tried symlinks within the gluster volume directory tree as well as symlinks to different file systems. >From a Windows client ALL symlinks appear to be files without extensions. On Linux clients the symlinks show as proper files/directories only when the symlinks are to another location on the gluster volume. If the symlinks are to a different file system samba clients list nothing. I've tested a non-VFS share [test] and symlinks are working ok. # testparm Load smb config files from /etc/samba/smb.conf rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) Processing section "[home]" Processing section "[test]" Loaded services file OK. Server role: ROLE_DOMAIN_MEMBER Press enter to see a dump of your service definitions [global] workgroup = DOMAIN realm = DOMAIN.CA server string = Samba Server Version %v security = ADS dedicated keytab file = /etc/krb5.keytab log file = /var/log/samba/log.%m max log size = 50 unix extensions = No load printers = No printcap name = /dev/null disable spoolss = Yes template homedir = /home/%U template shell = /bin/bash winbind use default domain = Yes winbind refresh tickets = Yes winbind offline logon = Yes idmap config DOMAIN:range = 4000-30 idmap config DOMAIN:schema_mode = rfc2307 idmap config DOMAIN:backend = ad idmap config * : range = 1000-2000 idmap config * : backend = tdb hosts allow = 127., <>.<>.194.96/27, <>.<>.230.32/27, 10.0.231.0/24 printing = bsd cups options = raw print command = lpr -r -P'%p' %s lpq command = lpq -P'%p' lprm command = lprm -P'%p' %j wide links = Yes [home] comment = exporting gluster filesystem via gfapi path = /home/ write list = +users force group = users read only = No create mask = 0660 force directory mode = 0770 kernel share modes = No vfs objects = glusterfs glusterfs:loglevel = 7 glusterfs:logfile = /var/log/samba/glusterfs-test.%M.log glusterfs:volume = storage glusterfs:volfile_server = 10.0.231.50 [test] path = /mnt/test/ write list = +users force group = users read only = No create mask = 0660 force directory mode = 0770 I can understand if samba VFS can't link to other filesystems, and I can work around this, but symlinks on the same gluster volume should work. I'll open a bug if someone can't see a config issue. I should mention that smb.conf contains 'follow symlinks = yes' but doesn't show in testparm, perhaps this is superceeded by 'wide links = yes'? OS = Centos 7.1 # rpm -qa | egrep -i 'gluster|samba' samba-4.1.12-24.el7_1.x86_64 glusterfs-cli-3.6.7-1.el7.x86_64 samba-common-4.1.12-24.el7_1.x86_64 glusterfs-3.6.7-1.el7.x86_64 glusterfs-fuse-3.6.7-1.el7.x86_64 samba-libs-4.1.12-24.el7_1.x86_64 samba-winbind-4.1.12-24.el7_1.x86_64 glusterfs-api-3.6.7-1.el7.x86_64 samba-vfs-glusterfs-4.1.12-24.el7_1.x86_64 samba-winbind-modules-4.1.12-24.el7_1.x86_64 glusterfs-libs-3.6.7-1.el7.x86_64 samba-winbind-clients-4.1.12-24.el7_1.x86_64 Thanks On Fri, Oct 2, 2015 at 4:42 PM, Steve Dainard wrote: > Hi Diego, > > Awesome, works - much appreciated. > > As far as I can search this isn't listed anywhere on the gluster.org > docs, but there is a link to a blog hosted here: > https://lalatendumohanty.wordpress.com/2014/02/11/using-glusterfs-with-samba-and-samba-vfs-plugin-for-glusterfs-on-fedora-20/ > > And it is also documented here: > https://www.mankier.com/8/vfs_glusterfs for future searches. > > On Thu, Oct 1, 2015 at 2:53 PM, Diego Remolina wrote: >> On all your shares where you use vfs objects = glusterfs also add the option: >> >> kernel share modes = No >> >> Then restart samba. >> >> Here is one of my example shares: >> >> [Projects] >>path = /projects >>browseable = yes >>write list = @Staff,root,@Admin,@Managers >>writeable = yes >>guest ok = no >>create mask = 660 >>directory mask = 770 >>kernel share modes = No >>vfs objects = glusterfs >>glusterfs:loglevel = 7 >>glusterfs:logfile = /var/log/samba/glusterfs-projects.log >>glusterfs:volume = export >> >> HTH, >> >> Diego >> >> On Thu, Oct 1, 2015 at 4:15 PM, Steve Dainard wrote: >>> samba-vfs-glusterfs-4.1.12-23.el7_1.x86_64 >>> gluster 3.6.6 >>> >>> I've shared a gluster volume using samba vfs with the options: >>> vfs objects = glusterfs >>> glusterfs:volume = test >>> path = / >>> >>> I can do the following: >>> (Windows client): >>> -Create new directory >>> -Create new file -- an error pops up "Unable to create the file 'New >>> Test Document.txt' The system cannot find the file specified.' BUT the >>> file is created anyways and shows up in the directory immediately >>> -Create a new sub directory in the above directory >>> -Delete or rename any file >>> (Linux client) >>> -Create new directory >>> -Create a new file under sub-directory -- "Device or resource busy" >>> BUT the file is created anyways - must refresh to see file in GUI >>> -Delete or rename any file >>> >>> Can not do: >>> (
[Gluster-users] geo-replication 3.6.7 -
Hi, we need some help...we are running a 6 node distributed-replicated master-volume which needs to be geo-replicated to another equal configured volume. recently both locations were updated from 3.4.7 to 3.5.6 to 3.6.7...running on ubuntu 14.04 lts. since update to 3.5.6 the geo-replication remains in status 'hybrid crawl' as shown below. [ 16:31:46 ] - root@gluster-ger-ber-09 /home/admin $gluster volume geo-replication ger-ber-01 gluster-wien-02::aut-wien-01 status detail MASTER NODE MASTER VOLMASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS FILES SYNCDFILES PENDINGBYTES PENDING DELETES PENDINGFILES SKIPPED --- gluster-ger-ber-09ger-ber-01/gluster-export gluster-wien-05-int::aut-wien-01Active N/A Changelog Crawl214645 402 11 gluster-ger-ber-12ger-ber-01/gluster-export gluster-wien-06-int::aut-wien-01Passive N/A N/A0 000 0 gluster-ger-ber-11ger-ber-01/gluster-export gluster-wien-03-int::aut-wien-01Active N/A Hybrid Crawl 2831091 000 491 gluster-ger-ber-10ger-ber-01/gluster-export gluster-wien-02-int::aut-wien-01Passive N/A N/A0 000 0 gluster-ger-ber-07ger-ber-01/gluster-export gluster-wien-07-int::aut-wien-01Active N/A Hybrid Crawl 2886662 000 474 gluster-ger-ber-08ger-ber-01/gluster-export gluster-wien-04-int::aut-wien-01Passive N/A N/A0 000 0 [ 16:31:51 ] - root@gluster-ger-ber-09 /home/admin $ on a slave we found a lot of repeating messages like : root@gluster-wien-07 /var/log/glusterfs/geo-replication-slaves $tail -f 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671\:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log ... [2015-12-01 23:30:19.782137] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-2: remote operation failed: File exists. Path: /rfvg2cmFNJ8Xt9HG.png [2015-12-01 23:30:19.782178] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-3: remote operation failed: File exists. Path: /rfvg2cmFNJ8Xt9HG.png [2015-12-01 23:30:19.782573] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 258115: SETXATTR() /.gfid/135991e9-5591-4217-b932-04cce065982f => -1 (File exists) ... all mentioned files have the same structure on the master volume...they appear on two sub-volumes. as you can see below, on one subvolume the mentioned file was created on the 8th jun. on that day we had a rebalance on the master-volume. I'm not really sure if this dht link file should have 0 bytes on this subvolume ? i already read this but i believe our situation is a bit more complicated... https://www.gluster.org/pipermail/gluster-users/2015-February/020608.html is it safe to delete those link files and the corresponding gfid on the affected bricks ? best regards dietmar tron@dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png master_gfid_file.txt e95a43c2-4969-4662-8ea9-72e02e82df90 thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png tron@dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-* ls-lisa-gluster.gluster-ger-ber-07-int.out:121247406422 612 -rw-rw-rw- 2 root root 619294 Okt 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-08-int.out:121289252163 612 -rw-rw-rw- 2 root root 619294 Okt 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-11-int.out:43282956834 4 -T 2 root root 619294 Jun 8 09:52 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-12-int.out:65271376731 4 -T 2 root root 619294 Jun 8 09:52 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png tron@dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-wien-0* ls-lisa-gluster.gluster-wien-04.out:82283414551 612 -rw-rw-rw- 2 root root 619294 Oct 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-wien-05.out:82278502871 612 -rw-rw-rw- 2 root root 619294 Oct 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png tron@dp-server:~/gluster-9$ -- Dietmar Putz 3Q Medien GmbH Wetzlarer Str. 86 D-14482 Potsdam Telefax: +49 (0)331 / 2797 866 - 1 Telefon: +49 (0)331 / 2792 866 - 8 Mobile: +49 171 / 90 160 39 Mail: p...@3qmedien.net ___ Glust
[Gluster-users] glusterfs-3.6.7 released
Hi, glusterfs-3.6.7 has been released and the packages for RHEL/Fedora/Centos can be found here.http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/ Requesting people running 3.6.x to please try it out and let us know if there are any issues. This release supposedly fixes the bugs listed below since 3.6.6 was made available. Thanks to all who submitted patches, reviewed the changes. 1283690 - core dump in protocol/client:client_submit_request 1283144 - glusterfs does not register with rpcbind on restart 1277823 - [upgrade] After upgrade from 3.5 to 3.6, probing a new 3.6 node is moving the peer to rejected state 1277822 - glusterd: probing a new node(>=3.6) from 3.5 cluster is moving the peer to rejected state Regards, Raghavendra Bhat ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
