[Gluster-users] What is the meaning of default 0 for performance.cache-max-file-size in glusterfs?

[Jin Li]

Hi all,

When I checked my glusterfs configuration, the default value of
performance.cache-max-file-size is 0. Please find the following
command that I have run.

$ sudo gluster volume set help
Option: performance.cache-max-file-size
Default Value: 0
Description: Maximum file size which would be cached by the io-cache translator.

Could you help show me what is the meaning of default value 0 for
performance.cache-max-file-size?

Please find my gluster version as below.

$ sudo gluster --version
glusterfs 3.6.9 built on Mar  2 2016 18:21:17
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. 
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU
General Public License.

Thanks for your help.

Best regards,
Jin
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] New commands for supporting add/remove brick and rebalance on tiered volume

[Hari Gowtham]

Hi,

Currently there are two suggested options for the syntax of add/remove brick:

1) gluster v tier  add-brick [replica ] [tier-type ] 
 ...

this syntax shows that its a add-brick operation on a tiered volume through a 
argument 
instead of distinguishing using the command. The separation of tier-type is 
done through 
parsing. When it comes to the parsing of these [replica ] [tier-type 
] ,
 we need to parse between the tier-type, replica count and bricks. All these 
three variable 
 make it complicated to parse and get the replica count, tier-type and brick. 

currently the parsing is like:
w = str_getunamb (words[3], opwords_cl);
if (!w) {
type = GF_CLUSTER_TYPE_NONE;
.
.
} else if ((strcmp (w, "replica")) == 0) { 
type = GF_CLUSTER_TYPE_REPLICATE;
.
.
}
} else if ((strcmp (w, "stripe")) == 0) { 
type = GF_CLUSTER_TYPE_STRIPE;
.
. 
} else {
.
. 
}

we can use the same for replica as long as replica comes before tier-type on 
the syntax.
and add the parsing for tier-type using words[4] instead of words[3] and repeat 
the same.
If its a plain distribute then we will be getting tier-type on words[3]. so we 
have to parse
it again by checking on the wordcount. the word count influences the parsing to 
a great extent.
Having the tier-type after replica is looking bit off as tier-type is more 
important here. 
So we can have tier-type before replica count. This has to be maintained 
thorughtout. 
And a separate parsing can make this work. Both these will influence the 
brick_index used 
for parsing the brick making the switch using the word_count bit unclear.
This can be done but will add a lot of complications on code.

2) gluster v tier  add-hot-brick/add-cold-brick [replica ] 
 ...
In this syntax, we remove the tier-type from parsing and mention the type on 
the command.
The parsing remains the same as add-brick parsing. as differentiate between the 
hot and cold 
brick is done by the command 

if (!strcmp(words[1], "detach-tier")) {
ret = do_cli_cmd_volume_detach_tier (state, word,
 words, wordcount);
goto out; 

} else if (!strcmp(words[1], "attach-tier")) {
ret = do_cli_cmd_volume_attach_tier (state, word,
 words, wordcount);
goto out; 
} else if (!strcmp(words[3], "add-hot-brick")) {

ret = do_cli_cmd_volume_add_hotbr_tier (state, word,
 words, wordcount-1);
goto out; 
} else if (!strcmp(words[3], "add-cold-brick")) {

ret = do_cli_cmd_volume_add_coldbr_tier (state, word,
 words, wordcount-1);
goto out; 
}

it get differentiated here and is sent to the respective function. and the 
parsing remains same. 

Let me know which one is the better one to follow.

- Original Message -
> From: "Hari Gowtham" 
> To: "Atin Mukherjee" 
> Cc: "gluster-users" , "gluster-devel" 
> 
> Sent: Monday, October 3, 2016 4:11:40 PM
> Subject: Re: [Gluster-devel] New commands for supporting add/remove brick and 
> rebalance on tiered volume
> 
> Yes. this sounds better than having two separate commands for each tier.
> If i don't get any other better solution will go with this one.
> Thanks Atin.
> 
> - Original Message -
> > From: "Atin Mukherjee" 
> > To: "Hari Gowtham" 
> > Cc: "gluster-devel" , "gluster-users"
> > 
> > Sent: Monday, October 3, 2016 4:02:42 PM
> > Subject: Re: [Gluster-devel] New commands for supporting add/remove brick
> > and rebalance on tiered volume
> > 
> > Hari,
> > 
> > I think you misunderstood my statement, probably I shouldn't have mentioned
> > existing semantics. One eg here should clarify it, so this is what I
> > propose:
> > 
> > gluster v tier  remove-brick tier-type hot  start
> > 
> > Note that my request was to add an argument i.e tier-type here.
> > 
> > 
> > On Monday 3 October 2016, Hari Gowtham  wrote:
> > 
> > > Hi Atin,
> > > Yes, we can do it. the existing semantics need some changes because of
> > > the
> > > attach tier command (gluster volume tier  attach ...) the
> > > parsing has to be changed to accommodate the attach tier command. if used
> > > as I
> > > mentioned then we can use the functions of attach tier generic for adding
> > > brick
> > > also. Other thing with using args is. it needs changes to support the
> > > keywords
> > 

[Gluster-users] EBADF after add-brick/self-heal operation in Gluster 3.7.15

[Rama Shenai]

Hi Gluster Team,

We saw a bunch of  intermittent EBADF errors on clients, We saw these
errors immediately after an add-brick operation followed by a self-heal of
that volume. We are wondering if these errors might close file descriptors
prematurely. causing problems on files we had open/memory-mapped

Below are the errors that we saw.  Any thoughts on this , as well as input
in avoiding this when we do live add-brick operations in the future is much
appreciated.

[2016-10-19 15:11:53.372930] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c003e7c
inode-gfid:b1e19a5b-7867-4cb3-8bf0-545df3c5d556) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373058] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c003e7c
inode-gfid:b1e19a5b-7867-4cb3-8bf0-545df3c5d556) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373105] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c0065b8
inode-gfid:9231d39d-d88c-4a62-b25d-fad232ec9b98) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373121] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c0065b8
inode-gfid:9231d39d-d88c-4a62-b25d-fad232ec9b98) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373138] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c005ac0
inode-gfid:a0b02209-59c9-418b-bff0-fb31be01b3e8) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373155] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c005ac0
inode-gfid:a0b02209-59c9-418b-bff0-fb31be01b3e8) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373172] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c004e18
inode-gfid:c490e1fe-3ac8-4c11-9c62-fc8672a27737) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373199] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c004e18
inode-gfid:c490e1fe-3ac8-4c11-9c62-fc8672a27737) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373231] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c0037bc
inode-gfid:1aab19de-f4b1-47bf-8216-d174797ae64d) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373245] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c0037bc
inode-gfid:1aab19de-f4b1-47bf-8216-d174797ae64d) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373271] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c004b90
inode-gfid:5c3d5a39-26f0-4211-a2f0-59de33ea5ade) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)
[2016-10-19 15:11:53.373287] W [fuse-resolve.c:556:fuse_resolve_fd]
0-fuse-resolve: migration of basefd (ptr:0x7f6d5c004b90
inode-gfid:5c3d5a39-26f0-4211-a2f0-59de33ea5ade) did not complete, failing
fop with EBADF (old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2)

Volume information
$ sudo gluster volume info
Volume Name: volume1
Type: Replicate
Volume ID: 3bcca83e-2be5-410c-9a23-b159f570ee7e
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ip-172-25-2-91.us-west-1.compute.internal:/data/
glusterfs/volume1/brick1/brick
Brick2: ip-172-25-2-206.us-west-1.compute.internal:/data/
glusterfs/volume1/brick1/brick
Brick3: 
ip-172-25-33-75.us-west-1.compute.internal:/data/glusterfs/volume1/brick1/brick
 <-- brick added
Options Reconfigured:
cluster.quorum-type: fixed
cluster.quorum-count: 2


Client translator configuration (from the client log)
  1: volume volume1-client-0
  2: type protocol/client
  3: option ping-timeout 42
  4: option remote-host ip-172-25-2-91.us-west-1.compute.internal
  5: option remote-subvolume /data/glusterfs/volume1/brick1/brick
  6: option transport-type socket
  7: option send-gids true
  8: end-volume
  9:
 10: volume volume1-client-1
 11: type protocol/client
 12: option ping-timeout 42
 13: option remote-host ip-172-25-2-206.us-west-1.compute.internal
 14: option remote-subvolume /data/glusterfs/volume1/brick1/brick
 15: option transport-type socket
 16: 

[Gluster-users] remote operation failed erros on Glusterfs 3.7.15

[Rama Shenai]

Hi Gluster team & users,

We are seeing multiple instances of the following error: "remote operation
failed [No such file or directory]" on our gluster clients, and this has
affects cases where we have some files hosted and are opened/memory-mapped

We are seeing this error after we recently added another brick to a replica
2 gluster volume (A couple of days back), making it a volume supported by
three replicated bricks (we performed this operation a couple of days
ago).  Any information on this error would be useful. If needed we can
supply any of the client or brick logs.

12447146-[2016-10-21 14:50:07.806214] I [dict.c:473:dict_get]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x148)
[0x7f68a0cc5f68]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x284)
[0x7f68a0aada94]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac)
[0x7f68a7f30dbc] ) 6-dict: !this || key=system.posix_acl_default [Invalid
argument]
12447579-[2016-10-21 14:50:07.837879] I [dict.c:473:dict_get]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x148)
[0x7f68a0cc5f68]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x230)
[0x7f68a0aada40]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac)
[0x7f68a7f30dbc] ) 6-dict: !this || key=system.posix_acl_access [Invalid
argument]
12448011-[2016-10-21 14:50:07.837928] I [dict.c:473:dict_get]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x148)
[0x7f68a0cc5f68]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x284)
[0x7f68a0aada94]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac)
[0x7f68a7f30dbc] ) 6-dict: !this || key=system.posix_acl_default [Invalid
argument]
12448444:[2016-10-21 14:50:10.784317] W [MSGID: 114031]
[client-rpc-fops.c:3057:client3_3_readv_cbk] 6-volume1-client-1: remote
operation failed [No such file or directory]
12448608:[2016-10-21 14:50:10.784757] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-0: remote
operation failed [No such file or directory]
12448772:[2016-10-21 14:50:10.784763] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-1: remote
operation failed [No such file or directory]
12448936:[2016-10-21 14:50:10.785575] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-2: remote
operation failed [No such file or directory]
12449100-[2016-10-21 14:50:10.786208] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_txn] 6-volume1-replicate-0: Unreadable
subvolume -1 found with event generation 3 for gfid
10495074-82d0-4961-8212-5a4f32895f37. (Possible split-brain)
12449328:[2016-10-21 14:50:10.787439] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-2: remote
operation failed [No such file or directory]
12449492-[2016-10-21 14:50:10.788730] E [MSGID: 109040]
[dht-helper.c:1190:dht_migration_complete_check_task] 6-volume1-dht:
(null): failed to lookup the file on volume1-dht [Stale file handle]
12449677-[2016-10-21 14:50:10.788778] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 622070230: READ => -1
gfid=10495074-82d0-4961-8212-5a4f32895f37 fd=0x7f68951a75bc (Stale file
handle)
12449864:The message "W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-1: remote
operation failed [No such file or directory]" repeated 2 times between
[2016-10-21 14:50:10.784763] and [2016-10-21 14:50:10.789213]
12450100:[2016-10-21 14:50:10.790080] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-2: remote
operation failed [No such file or directory]
12450264:The message "W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-0: remote
operation failed [No such file or directory]" repeated 3 times between
[2016-10-21 14:50:10.784757] and [2016-10-21 14:50:10.791118]
12450500:[2016-10-21 14:50:10.791176] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 6-volume1-client-1: remote
operation failed [No such file or directory]
12450664-[2016-10-21 14:50:10.793395] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 622070238: READ => -1
gfid=10495074-82d0-4961-8212-5a4f32895f37 fd=0x7f68951a75bc (Stale file
handle)
12450851-[2016-10-21 14:50:11.036804] I [dict.c:473:dict_get]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x148)
[0x7f68a0cc5f68]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.15/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x230)
[0x7f68a0aada40]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac)
[0x7f68a7f30dbc] ) 6-dict: !this || key=system.posix_acl_access [Invalid
argument]
12451283-[2016-10-21 14:50:11.036889] I [dict.c:473:dict_get]

Re: [Gluster-users] gluster volume not mounted on boot

[Ville-Pekka Vainio]

> On 21 Oct 2016, at 19.52,  
>  wrote:
> 
> Will that work for samba mounts? (cifs) 

Just based on a Google search, I’d say yes.
https://wiki.archlinux.org/index.php/samba#As_mount_entry

We use the FUSE mounts and with those it works well.

-Ville-Pekka
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume not mounted on boot

[lindsay.mathieson]

Will that work for samba mounts? (cifs)

Sent from my Windows 10 phone

From: Ville-Pekka Vainio
Sent: Saturday, 22 October 2016 2:26
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] gluster volume not mounted on boot

Hi all,

On a system using systemd, you should also be able to use x-systemd.automount.
Just add 'noauto,x-systemd.automount’ to your fstab line and systemd should 
mount the gluster volume for you when it’s accessed.
The ‘noauto’ option means that autofs won’t touch the mount and systemd will 
have control over it.

-Ville-Pekka

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume not mounted on boot

[lindsay.mathieson]

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume not mounted on boot

[Ville-Pekka Vainio]

Hi all,

On a system using systemd, you should also be able to use x-systemd.automount.
Just add 'noauto,x-systemd.automount’ to your fstab line and systemd should 
mount the gluster volume for you when it’s accessed.
The ‘noauto’ option means that autofs won’t touch the mount and systemd will 
have control over it.

-Ville-Pekka

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

[Lindsay Mathieson]

On 21/10/2016 12:39 PM, Lindsay Mathieson wrote:

And now I have it all setup for logging etc I can't reproduce the error:(


Ah, figured out what I was doing different - started the heavy I/O 
before the add bricks and rebalance which reliably triggers a "volume 
rebalance: teststore1: failed: Another transaction is in progress for 
teststore1. Please try again after sometime"



If I start the I/O after the rebalance starts then things rapidly go 
pear shaped. However busy this saturday (Spouses birthday), so I'll 
probably get the test and logs sorted Sunday.


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Reliably mounting a gluster volume

[Kevin Lemonnier]

Hi,

As we were discussing in the "gluster volume not mounted on boot" you
should probably just go with AutoFS, not ideal but I don't see any
other reliable solutions.


On Fri, Oct 21, 2016 at 02:46:05PM +0200, Paul Boven wrote:
> Hi everyone,
> 
> For the past few days I've been experimenting with Gluster and systemd. 
> The issue I'm trying to solve is that my gluster servers always fail to 
> self-mount their gluster volume locally on boot. Apparently this is 
> because the mount happens right after glusterd has been started, but 
> before it is ready to serve the volume.
> 
> I'm doing a refresh of our internal gluster based KVM system, bringing 
> it to Ubuntu 16.04LTS. As the Ubuntu gluster package as shipped still 
> has this boot/mount issue, and to simplify things a bit, I've removed 
> all SystemV and Upstart that ships with the current Ubuntu Gluster 
> package, aiming for a systemd-only solution. Ubuntu 16.04LTS uses systemd.
> 
> The problem, in my opinion, stems from the fact that in the Unit file 
> for glusterd, it is declared as a 'forking' kind of service. This means 
> that as soon as the double fork happens, systemd has no option but to 
> consider the service as available, and continues with the rest of its 
> work. I try to delay the mounting of my /gluster by adding 
> "x-systemd.requires=glusterd.service" but for the reasons above, that 
> still causes the mount to happen immediately after glusterd has started, 
> and then the mount fails.
> 
> Is there a way for systemd to know when the gluster service is actually 
> able to service a mount request, so one can delay this step of the boot 
> process?
> 
> In the Unit file, I have:
> [Unit]
> Requires=rpcbind.service
> After=network.target rpcbind.service network-online.target
> 
> The curious thing is that, according to gluster.log, the gluster client 
> does find out on which hostnames the subvolumes are available. However, 
> it seems that talking to both the local (0-gv0-client-0) as remote 
> (0-gv0-client-1) fails. For the service on localhost, the error is 
> 'failed to get the port number for remote subvolume'. For the remote 
> volume, it is 'no route to host'. But at this stage, local networking 
> (which is fully static and on the same network) should already be up.
> 
> Some error messages during the mount:
> 
> [12:15:50.749137] E [MSGID: 114058] 
> [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-0: 
> failed to get the port number for remote subvolume. Please run 'gluster 
> volume status' on server to see if brick process is running.
> [12:15:50.749178] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 
> 0-gv0-client-0: disconnected from gv0-client-0. Client process will keep 
> trying to connect to glusterd until brick's port is available
> [12:15:53.679570] E [socket.c:2278:socket_connect_finish] 
> 0-gv0-client-1: connection to 10.0.0.3:24007 failed (No route to host)
> [12:15:53.679611] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 
> 0-gv0-replicate-0: All subvolumes are down. Going offline until atleast 
> one of them comes back up.
> 
> Once the machine has fully booted and I log in, simply typing 'mount 
> /gluster' always succeeds. I would really appreciate your help in making 
> this happening on boot without intervention.
> 
> Regards, Paul Boven.
> -- 
> Paul Boven  +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.eu
> VLBI - It's a fringe science
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Reliably mounting a gluster volume

[Paul Boven]

Hi everyone,

For the past few days I've been experimenting with Gluster and systemd. 
The issue I'm trying to solve is that my gluster servers always fail to 
self-mount their gluster volume locally on boot. Apparently this is 
because the mount happens right after glusterd has been started, but 
before it is ready to serve the volume.


I'm doing a refresh of our internal gluster based KVM system, bringing 
it to Ubuntu 16.04LTS. As the Ubuntu gluster package as shipped still 
has this boot/mount issue, and to simplify things a bit, I've removed 
all SystemV and Upstart that ships with the current Ubuntu Gluster 
package, aiming for a systemd-only solution. Ubuntu 16.04LTS uses systemd.


The problem, in my opinion, stems from the fact that in the Unit file 
for glusterd, it is declared as a 'forking' kind of service. This means 
that as soon as the double fork happens, systemd has no option but to 
consider the service as available, and continues with the rest of its 
work. I try to delay the mounting of my /gluster by adding 
"x-systemd.requires=glusterd.service" but for the reasons above, that 
still causes the mount to happen immediately after glusterd has started, 
and then the mount fails.


Is there a way for systemd to know when the gluster service is actually 
able to service a mount request, so one can delay this step of the boot 
process?


In the Unit file, I have:
[Unit]
Requires=rpcbind.service
After=network.target rpcbind.service network-online.target

The curious thing is that, according to gluster.log, the gluster client 
does find out on which hostnames the subvolumes are available. However, 
it seems that talking to both the local (0-gv0-client-0) as remote 
(0-gv0-client-1) fails. For the service on localhost, the error is 
'failed to get the port number for remote subvolume'. For the remote 
volume, it is 'no route to host'. But at this stage, local networking 
(which is fully static and on the same network) should already be up.


Some error messages during the mount:

[12:15:50.749137] E [MSGID: 114058] 
[client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-0: 
failed to get the port number for remote subvolume. Please run 'gluster 
volume status' on server to see if brick process is running.
[12:15:50.749178] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 
0-gv0-client-0: disconnected from gv0-client-0. Client process will keep 
trying to connect to glusterd until brick's port is available
[12:15:53.679570] E [socket.c:2278:socket_connect_finish] 
0-gv0-client-1: connection to 10.0.0.3:24007 failed (No route to host)
[12:15:53.679611] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 
0-gv0-replicate-0: All subvolumes are down. Going offline until atleast 
one of them comes back up.


Once the machine has fully booted and I log in, simply typing 'mount 
/gluster' always succeeds. I would really appreciate your help in making 
this happening on boot without intervention.


Regards, Paul Boven.
--
Paul Boven  +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.eu
VLBI - It's a fringe science
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] failed to find key 'child_up' in the options

[Niels de Vos]

On Fri, Oct 21, 2016 at 12:14:15PM +0200, Josep Manel Andrés wrote:
> Hi again,
> I have two servers SLES 12 SP 0 with
> 
> Information for package glusterfs:
> --
> Repository: GlusterFS
> Name: glusterfs
> Version: 3.7.11-101.1
> Arch: x86_64
> Vendor: obs://build.opensuse.org/home:kkeithleatredhat
> 
> 
> and now I decided to upgrade one server to SP1 and install
> 
> Information for package glusterfs:
> --
> Repository: GlusterFS-3.8 (SLE_12_SP1)
> Name: glusterfs
> Version: 3.8.0-100.1
> Arch: x86_64
> Vendor: obs://build.opensuse.org/home:kkeithleatredhat
> 
> 
> but what happens now is that when trying to mount a volume with 3.8 it takes
> so long...over 6 or 7 seconds, and here is the moment where gets stacked in
> the logs:
> 
> [2016-10-21 10:05:18.285837] I [MSGID: 108005]
> [afr-common.c:4137:afr_notify] 0-volume1-replicate-0: Subvolume
> 'volume1-client-1' came back up; going online.
> [2016-10-21 10:05:18.285872] I [MSGID: 114035]
> [client-handshake.c:201:client_set_lk_version_cbk] 0-volume1-client-1:
> Server lk version = 1
> [2016-10-21 10:05:18.286023] W [MSGID: 114007]
> [client-handshake.c:1176:client_setvolume_cbk] 0-volume1-client-0: failed to
> find key 'child_up' in the options
> 
> 
> 
> [2016-10-21 10:05:29.273913] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse:
> switched to graph 0
> [2016-10-21 10:05:29.274151] I [fuse-bridge.c:4153:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
> 7.22
> [2016-10-21 10:05:29.275426] I [MSGID: 108031]
> [afr-common.c:1908:afr_local_discovery_cbk] 0-volume1-replicate-0: selecting
> local read_child volume1-client-1
> 
> 
> 
> Any idea about what happens? Should I upgrade de other server?

This looks related to https://bugzilla.redhat.com/1350326 . It has been
addressed in glusterfs-3.8.0, so I do not know if the delay in mounting
is expected. Without the fix mounting seemed to have been completely
non-functional. Avra should be able to explain the details a little
more.

Niels


signature.asc
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] failed to find key 'child_up' in the options

[Josep Manel Andrés]

Hi again,
I have two servers SLES 12 SP 0 with

Information for package glusterfs:
--
Repository: GlusterFS
Name: glusterfs
Version: 3.7.11-101.1
Arch: x86_64
Vendor: obs://build.opensuse.org/home:kkeithleatredhat


and now I decided to upgrade one server to SP1 and install

Information for package glusterfs:
--
Repository: GlusterFS-3.8 (SLE_12_SP1)
Name: glusterfs
Version: 3.8.0-100.1
Arch: x86_64
Vendor: obs://build.opensuse.org/home:kkeithleatredhat


but what happens now is that when trying to mount a volume with 3.8 it 
takes so long...over 6 or 7 seconds, and here is the moment where gets 
stacked in the logs:


[2016-10-21 10:05:18.285837] I [MSGID: 108005] 
[afr-common.c:4137:afr_notify] 0-volume1-replicate-0: Subvolume 
'volume1-client-1' came back up; going online.
[2016-10-21 10:05:18.285872] I [MSGID: 114035] 
[client-handshake.c:201:client_set_lk_version_cbk] 0-volume1-client-1: 
Server lk version = 1
[2016-10-21 10:05:18.286023] W [MSGID: 114007] 
[client-handshake.c:1176:client_setvolume_cbk] 0-volume1-client-0: 
failed to find key 'child_up' in the options




[2016-10-21 10:05:29.273913] I [fuse-bridge.c:5241:fuse_graph_setup] 
0-fuse: switched to graph 0
[2016-10-21 10:05:29.274151] I [fuse-bridge.c:4153:fuse_init] 
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 
kernel 7.22
[2016-10-21 10:05:29.275426] I [MSGID: 108031] 
[afr-common.c:1908:afr_local_discovery_cbk] 0-volume1-replicate-0: 
selecting local read_child volume1-client-1




Any idea about what happens? Should I upgrade de other server?

Best regards.


--
Josep Manel Andrés (josep.and...@bsc.es)
Operations - Barcelona Supercomputing Center
C/ Jordi Girona, 31  http://www.bsc.es
08034 Barcelona, Spain Tel: +34-93-401 25 73
e-mail: syst...@bsc.es Fax: +34-93-413 77 21
---

WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume not mounted on boot

[Kevin Lemonnier]

> But I have a question, it is possible to mount a volume at /gluster-data 
> for example???
> 

No, the way AutoFS works it needs a directory just for itself, so you can't do 
that.
I think the simplest would be to use /mnt/autofs like us, and then do a 
symbolic link :

ln -s /mnt/autofs/gluster-data /gluster-data

That way you'd end up with pretty much what you want.

-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume not mounted on boot

[Josep Manel Andrés]

Hi Kevin,

Thank you for your help, I manage to set it up!

But I have a question, it is possible to mount a volume at /gluster-data 
for example???



I cannot do:

opsld04:~ # cat /etc/auto.master
#
# Sample auto.master file
# This is a 'master' automounter map and it has the following format:
# mount-point [map-type[,format]:]map [options]
# For details of the format look at auto.master(5).
#
/ /etc/auto.map.d/master.autofs



and then :


opsld04:~ # cat /etc/auto.map.d/master.autofs
gluster-data 	 
-fstype=glusterfs,defaults,_netdev,backupvolfile-server=other_node 
  gluster1:/volume1




in auto.master file I have to set up a path other than root.

Did you manage to mount it  just at the first level of root, hanging from /

Cheers.

On 20/10/16 11:00, Kevin Lemonnier wrote:

On Thu, Oct 20, 2016 at 10:47:49AM +0200, Josep Manel Andrés wrote:

Thanks Kevin,

hahah, you are right, it works for me now, but it may not work in the
future when running tomcat on it, tomcat may need files from gluster
before it is mounted.



Yep, took us a while to find a solution here tbh.


So, I will go for AutoFS.



I find AutoFS hard to configure, especially with gluster, so here is
an example :

mkdir /etc/auto.map.d /etc/auto.master.d /mnt/autofs


In /etc/auto.master.d/master.autofs :

   /mnt/autofs /etc/auto.map.d/master.autofs

In /etc/auto.map.d/master.autofs :

   applicatif
-fstype=glusterfs,defaults,_netdev,backupvolfile-server=other_node
localhost:/applicatif

With this, /mnt/autofs/applicatif will automatically mount the /applicatif 
volume.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users



--
Josep Manel Andrés (josep.and...@bsc.es)
Operations - Barcelona Supercomputing Center
C/ Jordi Girona, 31  http://www.bsc.es
08034 Barcelona, Spain Tel: +34-93-401 25 73
e-mail: syst...@bsc.es Fax: +34-93-413 77 21
---

WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] locking problem on 3.8.5

[Bernhard Duebi]

Hi Atin,
the node with the UUID=bb12d7e7-ded5-4d32-b294-8f5011f70afb is one of
the nodes I sent the logs
root@chglbcvtprd04:~# cat /var/lib/glusterd/glusterd.info
UUID=bb12d7e7-ded5-4d32-b294-8f5011f70afb
operating-version=30703
below the info you requested, but I don't know how relevant it is as
the machines have been rebooted since my last mail
root@chastcvtprd04:~# gluster peer status
Number of Peers: 1
 
Hostname:
chglbcvtprd04
Uuid: bb12d7e7-ded5-4d32-b294-8f5011f70afb
State: Peer in
Cluster (Connected)
Other names:
chglbcvtprd04.fpprod.corp
root@chastcvtpr
d04:~# glusterd statedump
root@chastcvtprd04:~# cat
/var/log/glusterfs/statedump.log
[2016-10-20 14:57:04.636547] I [MSGID:
100030] [glusterfsd.c:2454:main] 0-glusterd: Started running glusterd
version 3.8.5 (args: glusterd statedump)
[2016-10-20 14:57:04.636599] E
[MSGID: 17] [glusterfsd.c:578:create_fuse_mount] 0-glusterfsd: Not
a client process, not performing mount operation
root@chastcvtprd04:~#
root@chglbcvtprd04:~# gluster peer status
Number of Peers: 1
 
Hostname:
chastcvtprd04.fpprod.corp
Uuid: 82aef154-8444-46bb-9fd5-d7eaf4f0a6bc
Stat
e: Peer in Cluster (Connected)
Other names:
chastcvtprd04
root@chglbcvtprd
04:~# glusterd statedump
root@chglbcvtprd04:~# cat
/var/log/glusterfs/statedump.log
[2016-10-20 14:59:21.047747] I [MSGID:
100030] [glusterfsd.c:2454:main] 0-glusterd: Started running glusterd
version 3.8.5 (args: glusterd statedump)
[2016-10-20 14:59:21.047785] E
[MSGID: 17] [glusterfsd.c:578:create_fuse_mount] 0-glusterfsd: Not
a client process, not performing mount operation
root@chglbcvtprd04:~#
 
 
Hope this helps
Let me know if you need more info
Best Regards
Bernhard
On Don, 2016-10-20 at 10:02 +0530, Atin Mukherjee wrote:
> I had a chance to look at the logs from both the nodes. I could see a
> repetitive instance of the following logs (in both the nodes):
> 
> "Lock for Oracle_Legal_04 held by bb12d7e7-ded5-4d32-b294
> -8f5011f70afb"
> 
> What that means is node with UUID is the culprit. However I doubt
> that the logs you shared from the nodes have this UUID. Could you
> please help me with gluster peer status output? If you happen to find
> which node is having the UUID mentioned, could you take glusterd
> statedump and share it with us?
> 
> On Tue, Oct 18, 2016 at 2:11 PM, Bernhard Duebi 
> wrote:
> > Hello,
> > 
> > I'm running gluster 3.8.5 on Ubuntu 16.04. I have 2 nodes which
> > mirror
> > each other. There are 32 volumes and all have the same
> > configuration:
> > 
> > Type: Replicate
> > Number of Bricks: 1 x 2 = 2
> > Transport-type: tcp
> > Bricks:
> > Brick1: node01:/data/glusterfs/vol/disk/brick
> > Brick2: node02:/data/glusterfs/vol/disk/brick
> > Options Reconfigured:
> > diagn
> > ostics.count-fop-hits: on
> > diagnostics.latency-measurement: on
> > performance
> > .readdir-ahead: on
> > nfs.disable: on
> > auth.allow:
> > 127.0.0.1,10.11.12.21,10.11.12.22
> > 
> > 
> > Nagios runs every 5 mins for each volume
> > # gluster volume heal $vol info
> > # gluster volume status $vol detail
> > 
> > Diamond runs every minute
> > # gluster volume list
> > and then for every volume
> > # gluster volume profile $vol info cumulative --xml
> > 
> > this was running fine with Gluster 3.7 but since I upgraded to
> > 3.8.5 I
> > see a lot of problems with locking. After a reboot of both machines
> > everything is fine. But after a while gluster volume status gives
> > me
> > the following error:
> > 
> > Another transaction is in progress for $vol. Please try again after
> > sometime
> > 
> > The problem is, that the system never recovers, only rebooting the
> > machines helps. Ok, probably a restart of gluster would do to.
> > 
> > I attached the logfiles from both glusterd. Let me know if you need
> > more information.
> > 
> > Thanks
> > Bernhard
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> 
> ___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New style community meetings - No more status updates

[Kaushal M]

On Thu, Oct 20, 2016 at 8:09 PM, Amye Scavarda  wrote:
>
>
> On Thu, Oct 20, 2016 at 7:06 AM, Kaushal M  wrote:
>>
>> Hi All,
>>
>> Our weekly community meetings have become mainly one hour of status
>> updates. This just drains the life out of the meeting, and doesn't
>> encourage new attendees to speak up.
>>
>> Let's try and change this. For the next meeting lets try skipping
>> updates all together and instead just dive into the 'Open floor' part
>> of the meeting.
>>
>> Let's have the updates to the regular topics be provided by the
>> regular owners before the meeting. This could either be through
>> sending out emails to the mailing lists, or updates entered into the
>> meeting etherpad[1]. As the host, I'll make sure to link to these
>> updates when the meeting begins, and in the meeting minutes. People
>> can view these updates later in their own time. People who need to
>> provide updates on AIs, just update the etherpad[1]. It will be
>> visible from there.
>>
>> Now let's move why I addressed this mail to this large and specific
>> set of people. The people who have been directly addressed are the
>> owners of the regular topics. You all are expected, before the next
>> meeting, to either,
>>  - Send out an update on the status for the topic you are responsible
>> for to the mailing lists, and then link to it on the the etherpad
>>  - or, provide you updates directly in the etherpad.
>> Please make sure you do this without fail.
>> If you do have anything to discuss, add it to the "Open floor" section.
>> Also, if I've missed out anyone in the addressed list, please make
>> sure they get this message too.
>>
>> Anyone else who wants to share their updates, add it to the 'Other
>> updates' section.
>>
>> Everyone else, go ahead and add anything you want to ask to the "Open
>> floor" section. Ensure to have your name with the topic you add
>> (etherpad colours are not reliable), and attend the meeting next week.
>> When your topic comes up, you'll have the floor.
>>
>> I hope that this new format helps make our meetings more colourful and
>> lively.
>>
>> As always, our community meetings will be held every Wednesday at
>> 1200UTC in #gluster-meeting on Freenode.
>> See you all there.
>>
>> ~kaushal
>>
>> [1]: https://public.pad.fsfe.org/p/gluster-community-meetings
>
>
> I really like this idea and am all in favor of color + liveliness.
> Let's give this new format three weeks or so, and we'll review around
> November 9th to see if we like this experiment.
> Fair?
> -- amye

Sounds good to me.

>
> --
> Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New style community meetings - No more status updates

[Amye Scavarda]

On Thu, Oct 20, 2016 at 7:06 AM, Kaushal M  wrote:

> Hi All,
>
> Our weekly community meetings have become mainly one hour of status
> updates. This just drains the life out of the meeting, and doesn't
> encourage new attendees to speak up.
>
> Let's try and change this. For the next meeting lets try skipping
> updates all together and instead just dive into the 'Open floor' part
> of the meeting.
>
> Let's have the updates to the regular topics be provided by the
> regular owners before the meeting. This could either be through
> sending out emails to the mailing lists, or updates entered into the
> meeting etherpad[1]. As the host, I'll make sure to link to these
> updates when the meeting begins, and in the meeting minutes. People
> can view these updates later in their own time. People who need to
> provide updates on AIs, just update the etherpad[1]. It will be
> visible from there.
>
> Now let's move why I addressed this mail to this large and specific
> set of people. The people who have been directly addressed are the
> owners of the regular topics. You all are expected, before the next
> meeting, to either,
>  - Send out an update on the status for the topic you are responsible
> for to the mailing lists, and then link to it on the the etherpad
>  - or, provide you updates directly in the etherpad.
> Please make sure you do this without fail.
> If you do have anything to discuss, add it to the "Open floor" section.
> Also, if I've missed out anyone in the addressed list, please make
> sure they get this message too.
>
> Anyone else who wants to share their updates, add it to the 'Other
> updates' section.
>
> Everyone else, go ahead and add anything you want to ask to the "Open
> floor" section. Ensure to have your name with the topic you add
> (etherpad colours are not reliable), and attend the meeting next week.
> When your topic comes up, you'll have the floor.
>
> I hope that this new format helps make our meetings more colourful and
> lively.
>
> As always, our community meetings will be held every Wednesday at
> 1200UTC in #gluster-meeting on Freenode.
> See you all there.
>
> ~kaushal
>
> [1]: https://public.pad.fsfe.org/p/gluster-community-meetings
>

I really like this idea and am all in favor of color + liveliness.
Let's give this new format three weeks or so, and we'll review around
November 9th to see if we like this experiment.
Fair?
-- amye

-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users