Re: [Gluster-users] Issues in AFR and self healing

[Ravishankar N]

On 08/15/2018 11:07 PM, Pablo Schandin wrote:


I found another log that I wasn't aware of in 
/var/log/glusterfs/brick, that is te mount log, I confused the log 
files. In this file I see a lot of entries like this one:


[2018-08-15 16:41:19.568477] I [addr.c:55:compare_addr_and_update] 
0-/mnt/brick1/gv1: allowed = "172.20.36.10", received addr = 
"172.20.36.11"
[2018-08-15 16:41:19.568527] I [addr.c:55:compare_addr_and_update] 
0-/mnt/brick1/gv1: allowed = "172.20.36.11", received addr = 
"172.20.36.11"
[2018-08-15 16:41:19.568547] I [login.c:76:gf_auth] 0-auth/login: 
allowed user names: 7107ccfa-0ba1-4172-aa5a-031568927bf1
[2018-08-15 16:41:19.568564] I [MSGID: 115029] 
[server-handshake.c:793:server_setvolume] 0-gv1-server: accepted 
client from 
physinfra-hb2.xcade.net-21091-2018/08/15-16:41:03:103872-gv1-client-0-0-0 
(version: 3.1

2.6)
[2018-08-15 16:41:19.582710] I [MSGID: 115036] 
[server.c:527:server_rpc_notify] 0-gv1-server: disconnecting 
connection from 
physinfra-hb2.xcade.net-21091-2018/08/15-16:41:03:103872-gv1-client-0-0-0
[2018-08-15 16:41:19.582830] I [MSGID: 101055] 
[client_t.c:443:gf_client_unref] 0-gv1-server: Shutting down 
connection 
physinfra-hb2.xcade.net-21091-2018/08/15-16:41:03:103872-gv1-client-0-0-0


So I see a lot of disconnections, right? This might be why the self 
healing is triggered all the time?


Not necessarily. These disconnects could also be due to the glfsheal 
binary which is invoked when you run `gluster vol heal volname info` etc 
and do not cause heals. It would be better to check your client mount 
logs for disconnect messages like these:


[2018-08-16 03:59:32.289763] I [MSGID: 114018] 
[client.c:2285:client_rpc_notify] 0-testvol-client-4: disconnected from 
testvol-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available


If there are no disconnects and you are still seeing files undergoing 
heal, then you might want to check the brick logs to see if there are 
any write failures.

Thanks,
Ravi


Thanks!

Pablo.

Avature

Get Engaged to Talent



On 08/14/2018 09:15 AM, Pablo Schandin wrote:


Thanks for the info!

I cannot see any logs in the mount log besides one line every time it 
rotates


[2018-08-13 06:25:02.246187] I 
[glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 0-glusterfs: No change in 
volfile,continuing


But I did find in the glfsheal-gv1.log of the volumes some kind of 
server-client connection that was disconnected and now it connects 
using a different port. The block of log per each run is kind of long 
so I'm copying it into a pastebin.


https://pastebin.com/bp06rrsT

Maybe this has something to do with it?

Thanks!

Pablo.

On 08/11/2018 12:19 AM, Ravishankar N wrote:




On 08/10/2018 11:25 PM, Pablo Schandin wrote:


Hello everyone!

I'm having some trouble with something but I'm not quite sure of 
with what yet. I'm running GlusterFS 3.12.6 on Ubuntu 16.04. I have 
two servers (nodes) in the cluster in a replica mode. Each server 
has 2 bricks. As the servers are KVM running several VMs, one brick 
has some VMs locally defined in it and the second brick is the 
replicated from the other server. It has data but not actual 
writing is being done except for the replication.


                            Server 1                               
                  Server 2
Volume 1 (gv1): Brick 1 defined VMs (read/write) >            
      Brick 1 replicated qcow2 files
Volume 2 (gv2): Brick 2 replicated qcow2 files <-            
 Brick 2 defined VMs (read/write)


So, the main issue arose when I got a nagios alarm that warned 
about a file listed to be healed. And then it disappeared. I came 
to find out that every 5 minutes, the self heal daemon triggers the 
healing and this fixes it. But looking at the logs I have a lot of 
entries in the glustershd.log file like this:


[2018-08-09 14:23:37.689403] I [MSGID: 108026] 
[afr-self-heal-common.c:1656:afr_log_selfheal] 0-gv1-replicate-0: 
Completed data selfheal on 407bd97b-e76c-4f81-8f59-7dae11507b0c. 
sources=[0] sinks=1
[2018-08-09 14:44:37.933143] I [MSGID: 108026] 
[afr-self-heal-common.c:1656:afr_log_selfheal] 0-gv2-replicate-0: 
Completed data selfheal on 73713556-5b63-4f91-b83d-d7d82fee111f. 
sources=[0] sinks=1


The qcow2 files are being healed several times a day (up to 30 in 
occasions). As I understand, this means that a data heal occurred 
on file with gfid 407b... and 7371... in source to sink. Local 
server to replica server? Is it OK for the shd to heal files in the 
replicated brick that supposedly has no writing on it besides the 
mirroring? How does that work?


In AFR, for writes, there is no notion of local/remote brick. No 
matter from which client you write to the volume, it gets sent to 
both bricks. i.e. the replication is synchronous and real time.


How does afr replication work? The file with gfid 7371... is the 
qcow2 root disk of an owncloud server with 17GB of data. It does 
not seem to be that big 

Re: [Gluster-users] Gluster release 3.12.13 (Long Term Maintenance) Canceled for 10th of August, 2018

[Jiffin Tony Thottan]

Since the issue seems to critical and there is no longer lock is held on 
Master branch, I will try to do a 3.12 release ASAP.


--

Jiffin

On Tuesday 14 August 2018 05:48 PM, Nithya Balachandran wrote:

I agree as well. This is a bug that is impacting users.

On 14 August 2018 at 16:30, Ravishankar N > wrote:


+1

Considering that master is no longer locked, it would be nice if a
release can be made sooner.  Amar sent a missing back port [1]
which also fixes a mem leak issue on the client side. This needs
to go in too.
Regards,
Ravi

[1] https://review.gluster.org/#/c/glusterfs/+/20723/



On 08/14/2018 04:20 PM, lemonni...@ulrar.net
 wrote:

Hi,

That's actually pretty bad, we've all been waiting for the
memory leak
patch for a while now, an extra month is a bit of a nightmare
for us.

Is there no way to get 3.12.12 with that patch sooner, at
least ? I'm
getting a bit tired of rebooting virtual machines by hand
everyday to
avoid the OOM killer ..

On Tue, Aug 14, 2018 at 04:12:28PM +0530, Jiffin Tony Thottan
wrote:

Hi,

Currently master branch is lock for fixing failures in the
regression
test suite [1].

As a result we are not releasing the next minor update for
the 3.12 branch,

which falls on the 10th of every month.

The next 3.12 update would be around the 10th of
September, 2018.

Apologies for the delay to inform above details.

[1]

https://lists.gluster.org/pipermail/gluster-devel/2018-August/055160.html



Regards,

Jiffin

___
Gluster-users mailing list
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-replication stops after 4-5 hours

[Marcus Pedersén]

Hi again Sunny,
Just a bit curious if you find anything in the logs that is useful and can help 
me get the geo-replication running.

Many thanks in advance!

Regards
Marcus

Från: gluster-users-boun...@gluster.org  för 
Marcus Pedersén 
Skickat: den 13 augusti 2018 22:45
Till: Sunny Kumar
Kopia: gluster-users@gluster.org
Ämne: Re: [Gluster-users] Geo-replication stops after 4-5 hours

Hi Sunny,
Please find the enclosed mount logs for the two active mater nodes.
I cut them down to todays logs.

Thanks!

Marcus


Från: Sunny Kumar 
Skickat: den 13 augusti 2018 21:49
Till: Marcus Pedersén
Kopia: Kotresh Hiremath Ravishankar; gluster-users@gluster.org
Ämne: Re: [Gluster-users] Geo-replication stops after 4-5 hours

Hi Marcus,

Can you please share mount log from slave (You can find it at
"/var/log/glusterfs/geo-replication-slaves/hostname/mnt.log").

- Sunny
On Tue, Aug 14, 2018 at 12:48 AM Marcus Pedersén  wrote:
>
> Hi again,
>
> New changes in behaviour, both master master nodes that are active toggles to 
> failure and the logs repeat the same over and over again.
>
>
> Part of log, node1:
>
> [2018-08-13 18:24:44.701711] I [gsyncdstatus(worker 
> /urd-gds/gluster):276:set_active] GeorepStatus: Worker Status Change
> status=Active
> [2018-08-13 18:24:44.704360] I [gsyncdstatus(worker 
> /urd-gds/gluster):248:set_worker_crawl_status] GeorepStatus: Crawl Status 
> Changestatus=History Crawl
> [2018-08-13 18:24:44.705162] I [master(worker /urd-gds/gluster):1448:crawl] 
> _GMaster: starting history crawlturns=1 stime=(1523907056, 0)   
> entry_stime=Noneetime=1534184684
> [2018-08-13 18:24:45.717072] I [master(worker /urd-gds/gluster):1477:crawl] 
> _GMaster: slave's time  stime=(1523907056, 0)
> [2018-08-13 18:24:45.904958] E [repce(worker /urd-gds/gluster):197:__call__] 
> RepceClient: call failed   call=5919:140339726538560:1534184685.88 
> method=entry_opserror=GsyncdError
> [2018-08-13 18:24:45.905111] E [syncdutils(worker 
> /urd-gds/gluster):298:log_raise_exception] : execution of "gluster" 
> failed with ENOENT (No such file or directory)
> [2018-08-13 18:24:45.919265] I [repce(agent 
> /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
> [2018-08-13 18:24:46.553194] I [monitor(monitor):272:monitor] Monitor: worker 
> died in startup phase brick=/urd-gds/gluster
> [2018-08-13 18:24:46.561784] I [gsyncdstatus(monitor):243:set_worker_status] 
> GeorepStatus: Worker Status Change status=Faulty
> [2018-08-13 18:24:56.581748] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> [2018-08-13 18:24:56.655164] I [gsyncd(worker /urd-gds/gluster):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-08-13 18:24:56.655193] I [gsyncd(agent /urd-gds/gluster):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-08-13 18:24:56.655889] I [changelogagent(agent 
> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> [2018-08-13 18:24:56.664628] I [resource(worker 
> /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
> [2018-08-13 18:24:58.347415] I [resource(worker 
> /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and 
> slave established.duration=1.6824
> [2018-08-13 18:24:58.348151] I [resource(worker 
> /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally...
> [2018-08-13 18:24:59.463598] I [resource(worker 
> /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume 
> duration=1.1150
> [2018-08-13 18:24:59.464184] I [subcmds(worker 
> /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. 
> Acknowledging back to monitor
> [2018-08-13 18:25:01.549007] I [master(worker 
> /urd-gds/gluster):1534:register] _GMaster: Working dir
> path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
> [2018-08-13 18:25:01.549606] I [resource(worker 
> /urd-gds/gluster):1253:service_loop] GLUSTER: Register time 
> time=1534184701
> [2018-08-13 18:25:01.593946] I [gsyncdstatus(worker 
> /urd-gds/gluster):276:set_active] GeorepStatus: Worker Status Change
> status=Active
>
>
> Part of log, node2:
>
> [2018-08-13 18:25:14.554233] I [gsyncdstatus(monitor):243:set_worker_status] 
> GeorepStatus: Worker Status Change status=Faulty
> [2018-08-13 18:25:24.568727] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> [2018-08-13 18:25:24.609642] I [gsyncd(agent /urd-gds/gluster):297:main] 
> : Using session config file   
> 

Re: [Gluster-users] Question to utime feature for release 4.1.0

[Jim Kinney]

Is this 'or' of atime settings also in 3.10/3.12 versions?

If it's set off in gluster but on in mount will atime be updated?

On August 15, 2018 2:15:17 PM EDT, Kotresh Hiremath Ravishankar 
 wrote:
>Hi David,
>
>The feature is to provide consistent time attributes (atime, ctime,
>mtime)
>across replica set.
>The feature is enabled with following two options.
>
>gluster vol set  utime on
>gluster vol set  ctime on
>
>The features currently does not honour mount options related time
>attributes such as 'noatime'.
>So even though the volume is mounted with noatime, it will still update
>atime with this feature
>enabled.
>
>Thanks,
>Kotresh HR
>
>On Wed, Aug 15, 2018 at 3:51 PM, David Spisla 
>wrote:
>
>> Dear Gluster Community,
>> in the Chapter "Standalone" point 3 of the release notes for 4.1.0
>> https://docs.gluster.org/en/latest/release-notes/4.1.0/
>>
>> there is an introduction to the new utime feature. What kind of
>options
>> are not allowed if I want to mount a volume? There is
>"noatime,realatime"
>> mentioned. Does the second mean "relatime". I never heard of
>"realatime".
>> Is there any recommendation for the mount options?
>>
>> Regards
>> David Spisla
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
>-- 
>Thanks and Regards,
>Kotresh H R

-- 
Sent from my Android device with K-9 Mail. All tyopes are thumb related and 
reflect authenticity.___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] KVM lockups on Gluster 4.1.1

[Walter Deignan]

I am using gluster to host KVM/QEMU images. I am seeing an intermittent 
issue where access to an image will hang. I have to do a lazy dismount of 
the gluster volume in order to break the lock and then reset the impacted 
virtual machine.

It happened again today and I caught the events below in the client side 
logs. Any thoughts on what might cause this? It seemed to begin after I 
upgraded from 3.12.10 to 4.1.1 a few weeks ago.

[2018-08-14 14:22:15.549501] E [MSGID: 114031] 
[client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 2-gv1-client-4: remote 
operation failed [Invalid argument]
[2018-08-14 14:22:15.549576] E [MSGID: 114031] 
[client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 2-gv1-client-5: remote 
operation failed [Invalid argument]
[2018-08-14 14:22:15.549583] E [MSGID: 108010] 
[afr-lk-common.c:284:afr_unlock_inodelk_cbk] 2-gv1-replicate-2: 
path=(null) gfid=----: unlock failed on 
subvolume gv1-client-4 with lock owner d89caca92b7f [Invalid argument]
[2018-08-14 14:22:15.549615] E [MSGID: 108010] 
[afr-lk-common.c:284:afr_unlock_inodelk_cbk] 2-gv1-replicate-2: 
path=(null) gfid=----: unlock failed on 
subvolume gv1-client-5 with lock owner d89caca92b7f [Invalid argument]
[2018-08-14 14:52:18.726219] E [rpc-clnt.c:184:call_bail] 2-gv1-client-4: 
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc5e00 
sent = 2018-08-14 14:22:15.699082. timeout = 1800 for 10.35.20.106:49159
[2018-08-14 14:52:18.726254] E [MSGID: 114031] 
[client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 2-gv1-client-4: remote 
operation failed [Transport endpoint is not connected]
[2018-08-14 15:22:25.962546] E [rpc-clnt.c:184:call_bail] 2-gv1-client-5: 
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc4a6d 
sent = 2018-08-14 14:52:18.726329. timeout = 1800 for 10.35.20.107:49164
[2018-08-14 15:22:25.962587] E [MSGID: 114031] 
[client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 2-gv1-client-5: remote 
operation failed [Transport endpoint is not connected]
[2018-08-14 15:22:25.962618] W [MSGID: 108019] 
[afr-lk-common.c:601:is_blocking_locks_count_sufficient] 
2-gv1-replicate-2: Unable to obtain blocking inode lock on even one child 
for gfid:24a48cae-53fe-4634-8fb7-0254c85ad672.
[2018-08-14 15:22:25.962668] W [fuse-bridge.c:1441:fuse_err_cbk] 
0-glusterfs-fuse: 3715808: FSYNC() ERR => -1 (Transport endpoint is not 
connected)

Volume configuration -

Volume Name: gv1
Type: Distributed-Replicate
Volume ID: 66ad703e-3bae-4e79-a0b7-29ea38e8fcfc
Status: Started
Snapshot Count: 0
Number of Bricks: 5 x 2 = 10
Transport-type: tcp
Bricks:
Brick1: dc-vihi44:/gluster/bricks/megabrick/data
Brick2: dc-vihi45:/gluster/bricks/megabrick/data
Brick3: dc-vihi44:/gluster/bricks/brick1/data
Brick4: dc-vihi45:/gluster/bricks/brick1/data
Brick5: dc-vihi44:/gluster/bricks/brick2_1/data
Brick6: dc-vihi45:/gluster/bricks/brick2/data
Brick7: dc-vihi44:/gluster/bricks/brick3/data
Brick8: dc-vihi45:/gluster/bricks/brick3/data
Brick9: dc-vihi44:/gluster/bricks/brick4/data
Brick10: dc-vihi45:/gluster/bricks/brick4/data
Options Reconfigured:
cluster.min-free-inodes: 6%
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 1
user.cifs: off
cluster.choose-local: off
features.shard: on
cluster.server-quorum-ratio: 51%

-Walter Deignan
-Uline IT, Systems Architect___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Question to utime feature for release 4.1.0

[Kotresh Hiremath Ravishankar]

Hi David,

The feature is to provide consistent time attributes (atime, ctime, mtime)
across replica set.
The feature is enabled with following two options.

gluster vol set  utime on
gluster vol set  ctime on

The features currently does not honour mount options related time
attributes such as 'noatime'.
So even though the volume is mounted with noatime, it will still update
atime with this feature
enabled.

Thanks,
Kotresh HR

On Wed, Aug 15, 2018 at 3:51 PM, David Spisla  wrote:

> Dear Gluster Community,
> in the Chapter "Standalone" point 3 of the release notes for 4.1.0
> https://docs.gluster.org/en/latest/release-notes/4.1.0/
>
> there is an introduction to the new utime feature. What kind of options
> are not allowed if I want to mount a volume? There is "noatime,realatime"
> mentioned. Does the second mean "relatime". I never heard of "realatime".
> Is there any recommendation for the mount options?
>
> Regards
> David Spisla
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Thanks and Regards,
Kotresh H R
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Issues in AFR and self healing

[Pablo Schandin]

I found another log that I wasn't aware of in /var/log/glusterfs/brick, 
that is te mount log, I confused the log files. In this file I see a lot 
of entries like this one:


[2018-08-15 16:41:19.568477] I [addr.c:55:compare_addr_and_update] 
0-/mnt/brick1/gv1: allowed = "172.20.36.10", received addr = "172.20.36.11"
[2018-08-15 16:41:19.568527] I [addr.c:55:compare_addr_and_update] 
0-/mnt/brick1/gv1: allowed = "172.20.36.11", received addr = "172.20.36.11"
[2018-08-15 16:41:19.568547] I [login.c:76:gf_auth] 0-auth/login: 
allowed user names: 7107ccfa-0ba1-4172-aa5a-031568927bf1
[2018-08-15 16:41:19.568564] I [MSGID: 115029] 
[server-handshake.c:793:server_setvolume] 0-gv1-server: accepted client 
from 
physinfra-hb2.xcade.net-21091-2018/08/15-16:41:03:103872-gv1-client-0-0-0 
(version: 3.1

2.6)
[2018-08-15 16:41:19.582710] I [MSGID: 115036] 
[server.c:527:server_rpc_notify] 0-gv1-server: disconnecting connection 
from 
physinfra-hb2.xcade.net-21091-2018/08/15-16:41:03:103872-gv1-client-0-0-0
[2018-08-15 16:41:19.582830] I [MSGID: 101055] 
[client_t.c:443:gf_client_unref] 0-gv1-server: Shutting down connection 
physinfra-hb2.xcade.net-21091-2018/08/15-16:41:03:103872-gv1-client-0-0-0


So I see a lot of disconnections, right? This might be why the self 
healing is triggered all the time?


Thanks!

Pablo.

Avature

Get Engaged to Talent



On 08/14/2018 09:15 AM, Pablo Schandin wrote:


Thanks for the info!

I cannot see any logs in the mount log besides one line every time it 
rotates


[2018-08-13 06:25:02.246187] I 
[glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 0-glusterfs: No change in 
volfile,continuing


But I did find in the glfsheal-gv1.log of the volumes some kind of 
server-client connection that was disconnected and now it connects 
using a different port. The block of log per each run is kind of long 
so I'm copying it into a pastebin.


https://pastebin.com/bp06rrsT

Maybe this has something to do with it?

Thanks!

Pablo.

On 08/11/2018 12:19 AM, Ravishankar N wrote:




On 08/10/2018 11:25 PM, Pablo Schandin wrote:


Hello everyone!

I'm having some trouble with something but I'm not quite sure of 
with what yet. I'm running GlusterFS 3.12.6 on Ubuntu 16.04. I have 
two servers (nodes) in the cluster in a replica mode. Each server 
has 2 bricks. As the servers are KVM running several VMs, one brick 
has some VMs locally defined in it and the second brick is the 
replicated from the other server. It has data but not actual writing 
is being done except for the replication.


                            Server 1                               
              Server 2
Volume 1 (gv1): Brick 1 defined VMs (read/write) >            
      Brick 1 replicated qcow2 files
Volume 2 (gv2): Brick 2 replicated qcow2 files <-            
 Brick 2 defined VMs (read/write)


So, the main issue arose when I got a nagios alarm that warned about 
a file listed to be healed. And then it disappeared. I came to find 
out that every 5 minutes, the self heal daemon triggers the healing 
and this fixes it. But looking at the logs I have a lot of entries 
in the glustershd.log file like this:


[2018-08-09 14:23:37.689403] I [MSGID: 108026] 
[afr-self-heal-common.c:1656:afr_log_selfheal] 0-gv1-replicate-0: 
Completed data selfheal on 407bd97b-e76c-4f81-8f59-7dae11507b0c. 
sources=[0]  sinks=1
[2018-08-09 14:44:37.933143] I [MSGID: 108026] 
[afr-self-heal-common.c:1656:afr_log_selfheal] 0-gv2-replicate-0: 
Completed data selfheal on 73713556-5b63-4f91-b83d-d7d82fee111f. 
sources=[0]  sinks=1


The qcow2 files are being healed several times a day (up to 30 in 
occasions). As I understand, this means that a data heal occurred on 
file with gfid 407b... and 7371... in source to sink. Local server 
to replica server? Is it OK for the shd to heal files in the 
replicated brick that supposedly has no writing on it besides the 
mirroring? How does that work?


In AFR, for writes, there is no notion of local/remote brick. No 
matter from which client you write to the volume, it gets sent to 
both bricks. i.e. the replication is synchronous and real time.


How does afr replication work? The file with gfid 7371... is the 
qcow2 root disk of an owncloud server with 17GB of data. It does not 
seem to be that big to be a bottleneck of some sort, I think.


Also, I was investigating the directory tree in 
brick/.glusterfs/indices and I notices that both in xattrop and 
dirty I always have a file created named xattrop-xx and 
dirty-xx. I read that the xattrop file is like a parent file or 
handle to reference other files created there as hardlinks with gfid 
name for the shd to heal. Is the same case as the ones in the dirty dir?


Yes, before the write, the gfid gets captured inside dirty on all 
bricks. If the write is successful, it gets removed. In addition, if 
the write fails on one brick, the other brick will capture the gfid 
inside xattrop.


Any help will be greatly appreciated it. Thanks!

If frequent 

Re: [Gluster-users] ganesha.nfsd process dies when copying files

[Karli Sjöberg]

On Wed, 2018-08-15 at 13:42 +0800, Pui Edylie wrote:
> Hi Karli,
> 
> I think Alex is right in regards with the NFS version and state.
> 
> I am only using NFSv3 and the failover is working per expectation.

OK, so I've remade the test again and it goes like this:

1) Start copy loop[*]
2) Power off hv02
3) Copy loop stalls indefinitely

I have attached a snippet of the ctdb log that looks interesting but
doesn't say much to me execpt that something's wrong:)

[*]: while true; do mount -o vers=3 hv03v.localdomain:/data /mnt/; dd
if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress; rm -fv
/mnt/test.bin; umount /mnt; done

Thanks in advance!

/K

> 
> In my use case, I have 3 nodes with ESXI 6.7 as OS and setup 1x 
> gluster VM on each of the ESXI host using its local datastore.
> 
> Once I have formed the replicate 3, I use the CTDB VIP to present the
> NFS3 back to the Vcenter and uses it as a shared storage.
> 
> Everything works great other than performance is not very good ... I
> am still looking for ways to improve it.
> 
> Cheers,
> Edy
> 
> On 8/15/2018 12:25 AM, Alex Chekholko wrote:
> > Hi Karli,
> > 
> > I'm not 100% sure this is related, but when I set up my ZFS NFS HA
> > per https://github.com/ewwhite/zfs-ha/wiki I was not able to get
> > the failover to work with NFS v4 but only with NFS v3.
> > 
> > From the client point of view, it really looked like with NFS v4
> > there is an open file handle and that just goes stale and hangs, or
> > something like that, whereas with NFSv3 the client retries and
> > recovers and continues.  I did not investigate further, I just use
> > v3.  I think it has something to do with NFSv4 being "stateful" and
> > NFSv3 being "stateless".
> > 
> > Can you re-run your test but using NFSv3 on the client mount?  Or
> > do you need to use v4.x?
> > 
> > Regards,
> > Alex
> > 
> > On Tue, Aug 14, 2018 at 6:11 AM Karli Sjöberg 
> > wrote:
> > > On Fri, 2018-08-10 at 09:39 -0400, Kaleb S. KEITHLEY wrote:
> > > > On 08/10/2018 09:23 AM, Karli Sjöberg wrote:
> > > > > On Fri, 2018-08-10 at 21:23 +0800, Pui Edylie wrote:
> > > > > > Hi Karli,
> > > > > > 
> > > > > > Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
> > > > > > 
> > > > > > I just installed them last weekend ... they are working
> > > very well
> > > > > > :)
> > > > > 
> > > > > Okay, awesome!
> > > > > 
> > > > > Is there any documentation on how to do that?
> > > > > 
> > > > 
> > > > https://github.com/gluster/storhaug/wiki
> > > > 
> > > 
> > > Thanks Kaleb and Edy!
> > > 
> > > I have now redone the cluster using the latest and greatest
> > > following
> > > the above guide and repeated the same test I was doing before
> > > (the
> > > rsync while loop) with success. I let (forgot) it run for about a
> > > day
> > > and it was still chugging along nicely when I aborted it, so
> > > success
> > > there!
> > > 
> > > On to the next test; the catastrophic failure test- where one of
> > > the
> > > servers dies, I'm having a more difficult time with.
> > > 
> > > 1) I start with mounting the share over NFS 4.1 and then proceed
> > > with
> > > writing a 8 GiB large random data file with 'dd', while "hard-
> > > cutting"
> > > the power to the server I'm writing to, the transfer just stops
> > > indefinitely, until the server comes back again. Is that supposed
> > > to
> > > happen? Like this:
> > > 
> > > # dd if=/dev/urandom of=/var/tmp/test.bin bs=1M count=8192
> > > # mount -o vers=4.1 hv03v.localdomain:/data /mnt/
> > > # dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
> > > 2434793472 bytes (2,4 GB, 2,3 GiB) copied, 42 s, 57,9 MB/s
> > > 
> > > (here I cut the power and let it be for almost two hours before
> > > turning
> > > it on again)
> > > 
> > > dd: error writing '/mnt/test.bin': Remote I/O error
> > > 2325+0 records in
> > > 2324+0 records out
> > > 2436890624 bytes (2,4 GB, 2,3 GiB) copied, 6944,84 s, 351 kB/s
> > > # umount /mnt
> > > 
> > > Here the unmount command hung and I had to hard reset the client.
> > > 
> > > 2) Another question I have is why some files "change" as you copy
> > > them
> > > out to the Gluster storage? Is that the way it should be? This
> > > time, I
> > > deleted eveything in the destination directory to start over:
> > > 
> > > # mount -o vers=4.1 hv03v.localdomain:/data /mnt/
> > > # rm -f /mnt/test.bin
> > > # dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
> > > 8557428736 bytes (8,6 GB, 8,0 GiB) copied, 122 s, 70,1 MB/s
> > > 8192+0 records in
> > > 8192+0 records out
> > > 8589934592 bytes (8,6 GB, 8,0 GiB) copied, 123,039 s, 69,8 MB/s
> > > # md5sum /var/tmp/test.bin 
> > > 073867b68fa8eaa382ffe05adb90b583  /var/tmp/test.bin
> > > # md5sum /mnt/test.bin 
> > > 634187d367f856f3f5fb31846f796397  /mnt/test.bin
> > > # umount /mnt
> > > 
> > > Thanks in advance!
> > > 
> > > /K
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > 

[Gluster-users] Question to utime feature for release 4.1.0

[David Spisla]

Dear Gluster Community,
in the Chapter "Standalone" point 3 of the release notes for 4.1.0
https://docs.gluster.org/en/latest/release-notes/4.1.0/

there is an introduction to the new utime feature. What kind of options are
not allowed if I want to mount a volume? There is "noatime,realatime"
mentioned. Does the second mean "relatime". I never heard of "realatime".
Is there any recommendation for the mount options?

Regards
David Spisla
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hello again :-)

The self heal must have finished as there are no log entries in
glustershd.log files anymore. According to munin disk latency (average
io wait) has gone down to 100 ms, and disk utilization has gone down
to ~60% - both on all servers and hard disks.

But now system load on 2 servers (which were in the good state)
fluctuates between 60 and 100; the server with the formerly failed
disk has a load of 20-30.I've uploaded some munin graphics of the cpu
usage:

https://abload.de/img/gluster11_cpu31d3a.png
https://abload.de/img/gluster12_cpu8sem7.png
https://abload.de/img/gluster13_cpud7eni.png

This can't be normal. 2 of the servers under heavy load and one not
that much. Does anyone have an explanation of this strange behaviour?


Thx :-)

2018-08-14 9:37 GMT+02:00 Hu Bert :
> Hi there,
>
> well, it seems the heal has finally finished. Couldn't see/find any
> related log message; is there such a message in a specific log file?
>
> But i see the same behaviour when the last heal finished: all CPU
> cores are consumed by brick processes; not only by the formerly failed
> bricksdd1, but by all 4 brick processes (and their threads). Load goes
> up to > 100 on the 2 servers with the not-failed brick, and
> glustershd.log gets filled with a lot of entries. Load on the server
> with the then failed brick not that high, but still ~60.
>
> Is this behaviour normal? Is there some post-heal after a heal has finished?
>
> thx in advance :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users