Re: [Gluster-users] Problem with massive file renaming in glusterfs volume

[Xavi Hernandez]

Hi Jose,

On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco 
wrote:

> Hi all,
>
> We have a problem with tasks (written in PHP) doing a lot of file
> renaming/moving (even several renames per second).
>
> The pattern is always the same: the task rename files with variable
> filename to the SAME final file (that is: the destination filename is
> ALWAYS the same).
>
> Problem: according to glusterfs and task logs, some rename are successful
> but other don't and we don't understand why or what is causing the error in
> the failing renames because the error is "File exists", but the rename
> supossedly must address these situations correctly and "overwrite" the
> destination file if it exists (it uses the PHP rename() function)
>
> More info:
>
> - We use a distributed-replicated volume in a 3 node cluster
>
> gluster volume info moodle-cv
>
> Volume Name: moodle-cv
> Type: Distributed-Replicate
> Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 5 x (2 + 1) = 15
> Transport-type: tcp
> Bricks:
> Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos
> Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica
> Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter)
> Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos
> Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica
> Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter)
> Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos
> Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica
> Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter)
> Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos
> Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica
> Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter)
> Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos
> Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica
> Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter)
> Options Reconfigured:
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> performance.open-behind: off
> performance.lazy-open: no
> cluster.self-heal-daemon: enable
>
> - The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5
>
> - The tasks use the PHP rename() function
>
> - The tasks renaming files are executed IN ONE NODE OF THE CLUSTER that
> mounts the volume using the FUSE client
>
> - At the end of the message I paste a (very) little fragment of the volume
> log
>
>
> Any ideas of the possible cause of the problem and/or suggestions to avoid
> it?
>

Most probably the issue is caused by special internal files needed by
Gluster that are used to reference the correct location of a file in a
distributed volume (they are referred as linkto files). I would say the
error happens because those files already exist when it's expected that
they don't exist (I think it's a similar problem as
https://github.com/gluster/glusterfs/issues/1723).

Are you using FUSE mounts ? are you doing the renames from more than one
client ?

Regards,

Xavi

Regards.
>
>
> This is the fragment of the log:
>
> * I've "separated" the lines by "rename operation" but all the lines
> appear together in the log
>
> --->SUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.478321] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (11fdf019-16f8-48ae-bd1f-05b670b29ec1)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)
>
> --->UNSUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.497845] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
> (350698df-35d9-4489-90cc-faa004bfc861)
> (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.503271] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-0: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.503339] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_c

Re: [Gluster-users] Announcing Gluster release 9.3 and GlusteFS SELinux 2.0.1

[Shwetha Acharya]

Hi Artem,

I have just built glusterfs 9.2 and 9.3 for Leap 15.2 as well.

Regards,
Shwetha


On Fri, Sep 3, 2021 at 4:51 AM Artem Russakovskii 
wrote:

> Hi,
>
> Can you please continue building gluster for OpenSUSE 15.2? It's still a
> supported and widely used version of OpenSUSE.
> https://en.opensuse.org/Lifetime
>
> The following distributions are expected to receive updates until the
>> specified date:
>>
>>- openSUSE Leap 15.3  - is
>>expected to be maintained until end of November 2022
>>
>>
>>- openSUSE Leap 15.2  - is
>>expected to be maintained until November
>>
>> 
>>  or
>>end of December 2021
>>
>>
> https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.3-9/openSUSE_Leap_15.3/x86_64/
> has 9.2 and 9.3 builds.
>
> https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.2-9/openSUSE_Leap_15.2/x86_64/
> only has 9.1.
>
> Thanks.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | @ArtemR 
>
>
> On Thu, Jul 15, 2021 at 5:37 AM Shwetha Acharya 
> wrote:
>
>> Hi All,
>>
>> The Gluster community is pleased to announce the release of GlusterFS 9.3
>> and GlusteFS SELinux 2.0.1
>>
>> GlusterFS 9.3:
>>
>> Packages available at [1].
>> Release notes for the release can be found at [2].
>>
>> Highlights of Release:
>>
>> - Core dumps on Gluster 9 - 3 replicas
>> - geo-rep: Improve handling of gfid mismatches
>> - auth.allow list is corrupted after add-brick (buffer overflow?)
>>
>>
>>
>> GlusterFS seLinux 2.0.1:
>>
>> Packages available at [3].
>> Release notes for the release can be found at [4].
>>
>> Highlights of Release:
>>
>> - glusterfs-selinux package should own the files created by it
>> - Fixing verification failure for ghost
>> - Adds rule to allow glusterd to access RDMA socket
>>
>>
>> Users are highly encouraged to upgrade to newer releases of GlusterFS and
>> Glusterfs seLinux at [1] and [3] respectively.
>>
>>
>> Thanks,
>> Shwetha
>>
>> References:
>>
>> [1] Packages for 9.3:
>> https://download.gluster.org/pub/gluster/glusterfs/9/9.3/
>>
>> [2] Release notes for 9.2:
>> https://docs.gluster.org/en/latest/release-notes/9.3/
>>
>> [3] Packages for glustefs-selinux2.0.1:
>> https://download.gluster.org/pub/gluster/glusterfs-selinux/2/2.0.1/
>>
>> [4] Release notes for glusterfs-selinux2.0.1:
>>
>> https://gluster.readthedocs.io/en/latest/release-notes/glusterfs-selinux2.0.1/
>> ---
>> 
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Glusterfs nfs mounts not showing directories

[John Cholewa]

My distributed volume had an issue on Friday which required a reboot
of the primary node. After this, I'm having a strange issue: When I
have the volume mounted via ganesha-nfs, using either the primary node
itself or a random workstation on the network, I'm seeing files from
both volumes, but I'm not seeing any directories at all. It's just a
listing of the files. But I *can* list the contents of a directory if
I know it exists. Similarly, that will show the files (in both nodes)
of that directory, but it will show no subdirectories. Example:

$ ls -F /mnt
flintstone/

$ ls -F /mnt/flintstone
test test1 test2 test3

$ ls -F /mnt/flintstone/wilma
file1 file2 file3

I've tried restarting glusterd on both nodes and rebooting the other
node as well. Mount options in fstab are defaults,_netdev,nofail. I
tried temporarily disabling the firewall in case that was a
contributing factor.

This has been working pretty well for over two years, and it's
survived system updates and reboots on the nodes, and there hasn't
been a recent software update that would have triggered this. The data
itself appears to be fine. 'gluster peer status' on each node shows
that the other is connected.

What's a good way to further troubleshoot this or to tell gluster to
figure itself out?  Would "gluster volume reset"  bring the
configuration to its original state without damaging the data in the
bricks?  Is there something I should look out for in the logs that
might give a clue?

Outputs:

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 18.04.4 LTS
Release:18.04
Codename:   bionic


# gluster --version
glusterfs 7.5
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. 
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.


# gluster volume status
Status of volume: gv0
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick yuzz:/gfs/brick1/gv0  N/A   N/AY   2909
Brick wum:/gfs/brick1/gv0   49152 0  Y   2885

Task Status of Volume gv0
--
There are no active volume tasks


# gluster volume info
Volume Name: gv0
Type: Distribute
Volume ID: dcfdeed9-8fe9-4047-b18a-1a908f003d7f
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: yuzz:/gfs/brick1/gv0
Brick2: wum:/gfs/brick1/gv0
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
features.cache-invalidation: on
cluster.readdir-optimize: off
performance.parallel-readdir: off
performance.cache-size: 8GB
network.inode-lru-limit: 100
performance.nfs.stat-prefetch: off


# gluster pool list
UUIDHostnameState
4b84240e-e73a-46da-9271-72f6001a8e18wum Connected
7de76707-cd99-4916-9c6b-ac6f26bda373localhost   Connected


Output of gluster get-state:
>
[Global]
MYUUID: 7de76707-cd99-4916-9c6b-ac6f26bda373
op-version: 31302

[Global options]

[Peers]
Peer1.primary_hostname: wum
Peer1.uuid: 4b84240e-e73a-46da-9271-72f6001a8e18
Peer1.state: Peer in Cluster
Peer1.connected: Connected
Peer1.othernames:

[Volumes]
Volume1.name: gv0
Volume1.id: dcfdeed9-8fe9-4047-b18a-1a908f003d7f
Volume1.type: Distribute
Volume1.transport_type: tcp
Volume1.status: Started
Volume1.brickcount: 2
Volume1.Brick1.path: yuzz:/gfs/brick1/gv0
Volume1.Brick1.hostname: yuzz
Volume1.Brick1.port: 0
Volume1.Brick1.rdma_port: 0
Volume1.Brick1.status: Started
Volume1.Brick1.spacefree: 72715274395648Bytes
Volume1.Brick1.spacetotal: 196003244277760Bytes
Volume1.Brick2.path: wum:/gfs/brick1/gv0
Volume1.Brick2.hostname: wum
Volume1.snap_count: 0
Volume1.stripe_count: 1
Volume1.replica_count: 1
Volume1.subvol_count: 2
Volume1.arbiter_count: 0
Volume1.disperse_count: 0
Volume1.redundancy_count: 0
Volume1.quorum_status: not_applicable
Volume1.snapd_svc.online_status: Offline
Volume1.snapd_svc.inited: True
Volume1.rebalance.id: ----
Volume1.rebalance.status: not_started
Volume1.rebalance.failures: 0
Volume1.rebalance.skipped: 0
Volume1.rebalance.lookedup: 0
Volume1.rebalance.files: 0
Volume1.rebalance.data: 0Bytes
Volume1.time_left: 0
Volume1.gsync_count: 0
Volume1.options.nfs.disable: on
Volume1.options.transport.address-family: inet
Volume1.options.features.cache-invalidation: on
Volume1.options.cluster.readdir-optimize: off
Volume1.options.performance.parallel-readdir: off
Volume1.options.performance.cache-size: 8GB
Volume1.options.network.inode-lru-limit: 100
Volume1.options.perform

[Gluster-users] gluster volume status show second node is offline

[Dario Lesca]

Hello everybody!
I'm a novice with gluster. I have setup my first cluster with two
nodes 

This is the current volume info:

   [root@s-virt1 ~]# gluster volume info gfsvol1
   Volume Name: gfsvol1
   Type: Replicate
   Volume ID: 5bad4a23-58cc-44d7-8195-88409720b941
   Status: Started
   Snapshot Count: 0
   Number of Bricks: 1 x 2 = 2
   Transport-type: tcp
   Bricks:
   Brick1: virt1.local:/gfsvol1/brick1
   Brick2: virt2.local:/gfsvol1/brick1
   Options Reconfigured:
   performance.client-io-threads: off
   nfs.disable: on
   transport.address-family: inet
   storage.fips-mode-rchecksum: on
   cluster.granular-entry-heal: on
   storage.owner-uid: 107
   storage.owner-gid: 107
   server.allow-insecure: on

For now all seem work fine.

I have mount the gfs volume on all two nodes and use the VM into it

But today I noticed that the second node (virt2) is offline:

   [root@s-virt1 ~]# gluster volume status
   Status of volume: gfsvol1
   Gluster process TCP Port  RDMA Port  Online  Pid
   
--
   Brick virt1.local:/gfsvol1/brick1   49152 0  Y   
3090 
   Brick virt2.local:/gfsvol1/brick1   N/A   N/AN   N/A 
 
   Self-heal Daemon on localhost   N/A   N/AY   
3105 
   Self-heal Daemon on virt2.local N/A   N/AY   
3140 

   Task Status of Volume gfsvol1
   
--
   There are no active volume tasks
   
   [root@s-virt1 ~]# gluster volume status gfsvol1 detail
   Status of volume: gfsvol1
   
--
   Brick: Brick virt1.local:/gfsvol1/brick1
   TCP Port : 49152   
   RDMA Port: 0   
   Online   : Y   
   Pid  : 3090
   File System  : xfs 
   Device   : /dev/mapper/rl-gfsvol1
   Mount Options: 
rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=128,noquota
   Inode Size   : 512 
   Disk Space Free  : 146.4GB 
   Total Disk Space : 999.9GB 
   Inode Count  : 307030856   
   Free Inodes  : 307026149   
   
--
   Brick: Brick virt2.local:/gfsvol1/brick1
   TCP Port : N/A 
   RDMA Port: N/A 
   Online   : N   
   Pid  : N/A 
   File System  : xfs 
   Device   : /dev/mapper/rl-gfsvol1
   Mount Options: 
rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=128,noquota
   Inode Size   : 512 
   Disk Space Free  : 146.4GB 
   Total Disk Space : 999.9GB 
   Inode Count  : 307052016   
   Free Inodes  : 307047307
   
What does it mean?
What's wrong?
Is this normal or I missing some setting?

If you need more information let me know

Many thanks for your help

-- 
Dario Lesca
(inviato dal mio Linux Fedora 34 Workstation)






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Recovering from remove-brick where shards did not rebalance

[Anthony Hoppe]

Hello,

I did a bad thing and did a remove-brick on a set of bricks in a 
distributed-replicate volume where rebalancing did not successfully rebalance 
all files.  In sleuthing around the various bricks on the 3 node pool, it 
appears that a number of the files within the volume may have been stored as 
shards.  With that, I'm unsure how to proceed with recovery.

Is it possible to re-add the removed bricks somehow and then do a heal?  Or is 
there a way to recover data from shards somehow?

Thanks!




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users