Re: [Gluster-users] Gluster Samba Options

[Anoop C S]

On Mon, 2020-02-10 at 14:50 +0100, Stefan Kania wrote:
> I have:
> root@cluster-02:~# ls /var/lib/glusterd/groups
> db-workload  gluster-block  metadata-cache  nl-cache  virt
> 
> and root@cluster-02:~# apt-file list glusterfs-server gives:
> ...
> glusterfs-server: /var/lib/glusterd/groups/db-workload
> glusterfs-server: /var/lib/glusterd/groups/gluster-block
> glusterfs-server: /var/lib/glusterd/groups/metadata-cache
> glusterfs-server: /var/lib/glusterd/groups/nl-cache
> glusterfs-server: /var/lib/glusterd/groups/virt
> ...
> No samba group :-(

Following group volume set option file for Samba seems to be missing
from Debian packaging. Can you please confirm whether it is the case or
not?

/var/lib/glusterd/groups/samba

> And looking with "dpkg -l | grep glusterfs-server gives:
> root@cluster-02:~# dpkg -l |grep glusterfs-server
> ii  glusterfs-server  7.2-1 amd64 clustered file-system (server
> package)
> seems to me the file is missing :-(
> 
> 
> Am 10.02.20 um 09:53 schrieb Anoop C S:
> > On Sun, 2020-02-09 at 15:44 +0100, Stefan Kania wrote:
> > > Am 08.02.20 um 11:33 schrieb Anoop C S:
> > > > # gluster volume set  group samba
> > > When i try to set the option I got the following error:
> > > Unable to open file '/var/lib/glusterd/groups/samba'. Error: No
> > > such
> > > file or directory
> > 
> > Do you have any other files under /var/lib/glusterd/groups/ ?
> > 
> > > And it's right there is no such file. I use Gluster 7.2-1 from
> > > the
> > > offical repository for debian buster. Am I missing a package? I
> > > just
> > > installeg glusterfs-server with it's dependencies
> 
> 



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Healing entries get healed but there are constantly new entries appearing

[Karthik Subrahmanya]

Hi Ulrich,

Thank you for letting us know. Glad to hear that your system is back to
normal.

Regards,
Karthik

On Mon, Feb 10, 2020 at 9:51 PM Ulrich Pötter 
wrote:

> Hello Karthik,
>
> thank you very much. That was exactly the problem.
> Running the command (cat
> /.meta/graphs/active/-client-*/private | egrep -i
> 'connected') on the clients revealed that a few were not connected to all
> bricks.
> After restarting them, everything went back to normal.
>
> Regards,
> Ulrich
> Am 06.02.20 um 12:51 schrieb Karthik Subrahmanya:
>
> Hi Ulrich,
>
> From the problem statement, seems like the client(s) have lost connection
> with brick. Can you give the following information?
> - How many clients are there for this volume and which version they are in?
> - gluster volume info  & gluster volume status  outputs
> - Check whether all the clients are connected to all the bricks.
> If you are using the fuse clients give the output of the following from
> all the clients
> cat /.meta/graphs/active/-client-*/private | egrep
> -i 'connected'
> -If you are using non fuse clients generate the statedumps (
> https://docs.gluster.org/en/latest/Troubleshooting/statedump/) of each
> clients and give the output of
> grep -A 2 "xlator.protocol.client" /var/run/gluster/
> (If you have changed the statedump-path replace the path in the above
> command)
>
> Regards,
> Karthik
>
> On Thu, Feb 6, 2020 at 5:06 PM Ulrich Pötter 
> wrote:
>
>> Dear Gluster Users,
>>
>> we are running the following Gluster setup:
>> Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and one was
>> upgraded to Centos 7.7 with Gluster 6.7.
>>
>> Since the upgrade to gluster 6.7 on one of the servers, we encountered
>> the following issue:
>> New healing entries appear and get healed, but soon afterwards new
>> healing entries appear.
>> The abovementioned problem started after we upgraded the server.
>> The healing issues do not only appear on the upgraded server, but on all
>> three.
>>
>> This does not seem to be a split brain issue as the output of the
>> command "gluster volume head  info split-brain" is "number of
>> entries in split-brain: 0"
>>
>> Has anyone else observed such behavior with different Gluster versions
>> in one replica setup?
>>
>> We hesitate with updating the other nodes, as we do not know if this
>> standard Gluster behaviour or if there is more to this problem.
>>
>> Can you help us?
>>
>> Thanks in advance,
>> Ulrich
>>
>> 
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/441850968
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] question on rebalance errors gluster 7.2 (adding to distributed/replicated)

[Erik Jacobson]

My question: Are the errors and anomalies below something I need to
investigate? Are should I not be worried?


I installed a test cluster to gluster 7.2 to run some tests, preparing
to see if we gain confidence to put this on the 5,120 node
supercomputer instead of gluster 4.1.6.

I started with a 3x2 volume with heavy optimizations for writes and NFS.
(6 nodes, distribute/replicate).

I booted my NFS-root clients and maintained them online.

I then performaned a add-brick operation to make it a 3x3 instead of
3.2 (so 9 servers instead of 6).

The rebalance went much better for me than gluster 4.1.6. However, I saw
some errors. We noted them first here -- 14 errors on leader8, and a few
on the others. These are the NEW nodes so the data flow was from the old
nodes to these three that at least have one error:

[root@leader8 glusterfs]# gluster volume rebalance cm_shared status
Node Rebalanced-files  size   
scanned  failures   skipped   status  run time in h:m:s
   -  ---   ---   
---   ---   ---  --
  leader1.head.cm.eag.rdlabs.hpecorp.net18933   596.4MB
181780 0  3760completed0:41:39
  172.23.0.418960 1.2GB
181831 0  3766completed0:41:39
  172.23.0.518691 1.2GB
181826 0  3716completed0:41:39
  172.23.0.614917   618.8MB
175758 0  3869completed0:35:40
  172.23.0.715114   573.5MB
175728 0  3853completed0:35:41
  172.23.0.814864   459.2MB
175742 0  3951completed0:35:40
  172.23.0.900Bytes 
   11 3 0completed0:08:26
 172.23.0.1100Bytes 
  242 1 0completed0:08:25
   localhost00Bytes 
514 0completed0:08:26
volume rebalance: cm_shared: success



My rebalance log is like 32M and I find it's hard for people to help me
when I post that much data. So I've tried to filter some of the data
here. Two classes -- anomalies and errors.


Errors (14 reported on this node):

[root@leader8 glusterfs]# grep -i "error from gf_defrag_get_entry" 
cm_shared-rebalance.log
[2020-02-10 23:23:55.286830] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:12.903496] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:15.226948] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:15.259480] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:15.398784] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:16.633033] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:16.645847] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:21.783528] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:24:22.307464] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:25:23.391256] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:26:34.203129] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:26:39.669243] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:27:42.615081] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry
[2020-02-10 23:28:53.942158] W [dht-rebalance.c:3439:gf_defrag_process_dir] 
0-cm_shared-dht: Found error from gf_defrag_get_entry


Brick log errors around 23:23:55 (to match the first error above):

[2020-02-10 23:23:54.605681] W [MSGID: 113096] 
[posix-handle.c:834:posix_handle_soft] 0-cm_shared-posix: symlink 
../../a4/3e/a43ef7fd-08eb-434c-8168-96a92059d186/LC_MESSAGES -> 

Re: [Gluster-users] NFS clients show missing files while gluster volume rebalanced

[Erik Jacobson]

Closing the loop in case someone does a search on this...

I have an update. I am getting some time on 1,000 node soon so I have
started to validate jumping to gluster 7.2 on my small lab machine.

I switched the packages to my own build of gluster 7.2 with gnfs.
I re-installed my leader node (gluster/gnfs servers) and created
the volumes the same way as before. This includes heavy cache
optimization for the NFS services volume.

I can no longer duplicate this problem on gluster 7.2. I was able to
duplicate rebalance troubles on NFS clients every time on gluster
4.1.6.

I do have a couple questions on some rebalance errors, which I will send
in a separate email.

Erik

On Wed, Jan 29, 2020 at 06:20:34PM -0600, Erik Jacobson wrote:
> We are using gluster 4.1.6. We are using gluster NFS (not ganesha).
> 
> Distributed/replicated with subvolume size 3 (6 total servers, 2
> subvols).
> 
> The NFS clients use this for their root filesystem.
> 
> When I add 3 more gluster servers to add one more subvolume to the
> storage volumes (so now subvolume size 3, 9 total servers, 3 total
> subvolumes), the process gets started. 
> 
> ssh leader1 gluster volume add-brick cm_shared 
> 172.23.0.9://data/brick_cm_shared 172.23.0.10://data/brick_cm_shared 
> 172.23.0.11://data/brick_cm_shared
> 
> then
> 
> ssh leader1 gluster volume rebalance cm_shared start
> 
> The re-balance works. 'gluster volume status' shows re-balance in
> progress.
> 
> However, existing gluster-NFS clients now show missing files and I can
> no longer log into them (since NFS is their root). If you are logged in,
> you can find that libraries are missing and general unhappiness with
> random files now missing.
> 
> Is accessing a volume that is in the process of being re-balanced not
> supported from a gluster NFS client? Or have I made an error?
> 
> Thank you for any help,
> 
> Erik



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Geo-replication /var/lib space question

[Alexander Iliev]

Hello list,

I have been running a geo-replication session for some time now, but at 
some point I noticed that the /var/lib/misc/gluster is eating up the 
storage on my root partition.


I moved the folder away to another partition, but I don't seem to 
remember reading any specific space requirement for /var/lib and 
geo-replication. Did I miss it in the documentation?


Also, does the space used in /var/lib/misc/gluster depend on the 
geo-replicated volume size? What exactly is stored there? (I'm guessing 
that's where gsyncd keeps track of the replicatation progress.)


(I'm running gluster 6.6 on CentOS 7.7.)

Thanks!
--
alexander iliev


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] It appears that readdir is not cached for FUSE mounts

[Strahil Nikolov]

On February 10, 2020 5:32:29 PM GMT+02:00, Matthias Schniedermeyer 
 wrote:
>On 10.02.20 16:21, Strahil Nikolov wrote:
>> On February 10, 2020 2:25:17 PM GMT+02:00, Matthias Schniedermeyer
> wrote:
>>> Hi
>>>
>>>
>>> I would describe our basic use case for gluster as:
>>> "data-store for a cold-standby application".
>>>
>>> A specific application is installed on 2 hardware machines, the data
>is
>>> kept in-sync between the 2 machines by a replica-2 gluster volume.
>>> (IOW: "RAID 1")
>>>
>>> At any one time only 1 machine has the volume mounted and the
>>> application running. If the machine goes down the application is
>>> started
>>> on the remaining machine.
>>> IOW at any one point in time there is only ever 1 "reader & writer"
>>> running.
>>>
>>> I profiled a performance problem we have with this application,
>which
>>> unfortunately we can't modify.
>>>
>>> The profile shows many "opendir/readdirp/releasedir" cycles, the
>>> directory in question has about 1000 files and the application
>"stalls"
>>> for several milliseconds any time it decides to do a readdir.
>>> The volume is mounted via FUSE and it appears that said operation is
>>> not
>>> cached at all.
>>>
>>> To provide a test-case i tried to replicate what the application
>does.
>>> The problematic operation is nearly perfectly emulated just by using
>>> "ls .".
>>>
>>> I created a script that replicates how we use gluster and
>demonstrates
>>> that a FUSE-mount appears to be lacking any caching of readdir.
>>>
>>> A word about the test-environment:
>>> 2 identical servers
>>> Dual Socket Xeon CPU E5-2640 v3 (8 cores, 2.60GHz, HT enabled)
>>> RAM: 128GB DDR4 ECC (8x16GB)
>>> Storage: 2TB Intel P3520 PCIe-NVMe-SSD
>>> Network: Gluster: 10GB/s direct connect (no switch), external:
>1Gbit/s
>>> OS: CentOS 7.7, Installed with "Minimal" ISO, everything: Default
>>> Up2Date as of: 2020-01-21 (Kernel: 3.10.0-1062.9.1.el7.x86_64)
>>> SELinux: Disabled
>>> SSH-Key for 1 -> 2 exchanged
>>> Gluster 6.7 packages installed via 'centos-release-gluster6'
>>>
>>> see attached:
>gluster-testcase-no-caching-of-dir-operations-for-fuse.sh
>>>
>>> The meat of the testcase is this:
>>> a profile of:
>>> ls .
>>> vs:
>>> ls . . . . . . . . . .
>>> (10 dots)
>>>
 cat /root/profile-1-times | grep DIR | head -n 3
>>> 0.00   0.00 us   0.00 us   0.00 us  1
>>> RELEASEDIR
>>> 0.27  66.79 us  66.79 us  66.79 us  1 
>OPENDIR
>>> 98.65   12190.30 us9390.88 us   14989.73 us  2
>>> READDIRP
>>>
 cat /root/profile-10-times | grep DIR | head -n 3
>>> 0.00   0.00 us   0.00 us   0.00 us 10
>>> RELEASEDIR
>>> 0.64 108.02 us  85.72 us 131.96 us 10 
>OPENDIR
>>> 99.368388.64 us5174.71 us   14808.77 us 20
>>> READDIRP
>>>
>>> This testcase shows perfect scaling.
>>> 10 times the request, results in 10 times the gluster-operations.
>>>
>>> I would say ideally there should be no difference in the number of
>>> gluster-operations, regardless of how often a directory is read in a
>>> short amount of time (with no changes in between)
>>>
>>>
>>> Is there something we can do to enable caching or otherwise improve
>>> performance?
>> 
>> Hi Matthias,
>> 
>> Have you tried the 'readdir-ahead' option .
>> According to docs it is useful for ' improving sequential directory
>read performance' .
>> I'm not sure how gluster defines sequential directory read, but it's
>worth trying.
>
>readdir-ahead is enabled by default. Has been for several years.
>In effect this option changes how many READDIRP OPs are executed for a
>single "ls .".
>(It takes more OPs when readdir-ahead is disabled.)
>
>> Also, you can try metadata caching , as described in:
>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/sect-directory_operations
>> The actual group should contain the following:
>>
>https://github.com/gluster/glusterfs/blob/master/extras/group-metadata-cache
>
>Metadata-Caching, in general, works:
>e.g. `stat FILE` is cached if executed repeatedly.
>
>AFAICT the big exception to metadata-caching is readdir.

Hi Matthias,

This now has turned into 'shoot into the dark'.

I have checked a nice presentation and these 2 attracted my attention:
performance.parallel-readdir on
cluster.readdir-hashed on

Presentation is found at:
https://www.google.com/url?sa=t=web=j=https://events.static.linuxfound.org/sites/events/files/slides/Gluster_DirPerf_Vault2017_0.pdf=2ahUKEwirh-qBs8fnAhWTTxUIHfn3CWEQFjAAegQIAhAB=AOvVaw1yhHZaWovhYGCexkGaMVQ8=1581352097024


I hope you find something useful there.

Best Regards,
Strahil Nikolov


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org

Re: [Gluster-users] Healing entries get healed but there are constantly new entries appearing

[Ulrich Pötter]

Hello Karthik,

thank you very much. That was exactly the problem.
Running the command (cat 
/.meta/graphs/active/-client-*/private | egrep -i 
'connected') on the clients revealed that a few were not connected to 
all bricks.

After restarting them, everything went back to normal.

Regards,
Ulrich

Am 06.02.20 um 12:51 schrieb Karthik Subrahmanya:

Hi Ulrich,

From the problem statement, seems like the client(s) have lost 
connection with brick. Can you give the following information?
- How many clients are there for this volume and which version they 
are in?
- gluster volume info  & gluster volume status  
outputs

- Check whether all the clients are connected to all the bricks.
If you are using the fuse clients give the output of the following 
from all the clients
cat /.meta/graphs/active/-client-*/private | 
egrep -i 'connected'
-If you are using non fuse clients generate the statedumps 
(https://docs.gluster.org/en/latest/Troubleshooting/statedump/) of 
each clients and give the output of

grep -A 2 "xlator.protocol.client" /var/run/gluster/
(If you have changed the statedump-path replace the path in the above 
command)


Regards,
Karthik

On Thu, Feb 6, 2020 at 5:06 PM Ulrich Pötter 
mailto:ulrich.poet...@menzel-it.net>> 
wrote:


Dear Gluster Users,

we are running the following Gluster setup:
Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and
one was
upgraded to Centos 7.7 with Gluster 6.7.

Since the upgrade to gluster 6.7 on one of the servers, we
encountered
the following issue:
New healing entries appear and get healed, but soon afterwards new
healing entries appear.
The abovementioned problem started after we upgraded the server.
The healing issues do not only appear on the upgraded server, but
on all
three.

This does not seem to be a split brain issue as the output of the
command "gluster volume head  info split-brain" is "number of
entries in split-brain: 0"

Has anyone else observed such behavior with different Gluster
versions
in one replica setup?

We hesitate with updating the other nodes, as we do not know if this
standard Gluster behaviour or if there is more to this problem.

Can you help us?

Thanks in advance,
Ulrich



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Permission denied at some directories/files after a split brain

[Strahil Nikolov]

On February 10, 2020 3:53:08 PM GMT+02:00, Alberto Bengoa  
wrote:
>Hello guys,
>
>We are running GlusterFS 6.6 in Replicate mode (1 x 3). After a
>split-brain
>and a massive heal process, we noticed that our app started to receive
>thousands of permissions denied while trying to access files and
>directories.
>
>Exemple log of a failed access atempt to a specific directory:
>
>[2020-02-10 10:38:17.402080] I [MSGID: 139001]
>[posix-acl.c:263:posix_acl_log_permit_denied]
>0-app_data-access-control:
>client:
>CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
>gfid: 092f1e28-d6a8-4ca9-95d5-75dc8ad1c835,
>req(uid:498,gid:498,perm:4,ngrps:1),
>ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
>[Permission denied]
>[2020-02-10 10:38:17.402182] E [MSGID: 115056]
>[server-rpc-fops_v2.c:687:server4_opendir_cbk] 0-app_data-server:
>6257941:
>OPENDIR /mailboxes.old/8692/211411002/Old
>(092f1e28-d6a8-4ca9-95d5-75dc8ad1c835), client:
>CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
>error-xlator: app_data-access-control [Permission denied]
>
>The permission denied happens only to unprivileged users, even if that
>unprivileged user is the directory owner. The root user is able to
>access
>all files, and if we "touch" the file/directory as root it *sometimes*
>fixes the problem.
>
>We noticed inconsistent Access/Change dates. Here a stat of a directory
>before touching it, showing these inconsistencies:
>
>  File: ‘Old’
>  Size: 4096   Blocks: 8  IO Block: 131072 directory
>Device: 27h/39d Inode: 10388898073370567318  Links: 2
>Access: (2775/drwxrwsr-x)  Uid: (  498/app)   Gid: (  498/app)
>Access: 1970-01-01 01:00:00.0 +0100
>Modify: 2020-02-07 13:21:10.365297527 +
>Change: 1970-01-01 01:00:00.0 +0100
> Birth: -
>
>I think this case is similar to the reported here[1] and discussed at
>thread "ACL issue v6.6, v6.7, v7.1, v7.2", despite the fact that we are
>not
>using libvirt. We do use ACLs, but not in this particular directory.
>
>Any thoughts on this?
>
>[1] - https://bugzilla.redhat.com/show_bug.cgi?id=1797099
>
>Thanks,
>Alberto Bengoa

Hi Alberto,
Sadly you should verify if the issue is the same.
Enable the trace logs for the bricks and verify if the errors in the logs with 
those in the bugzilla.
Don't forget to stop the trace log or your logs' dir will get full.

What version of gluster are you using ?
In my case only a  downgrade has restored the operation of the cluster, so you 
should consider that as an option (last, but still an option).

You can try to run a find against the fuse and 'find  /path/to/fuse -exec 
setfacl -m u:root:rw {} \;'
Maybe that will force gluster to read the ACLs again.

Good luck!
If you have the option, join the next gluster meeting and ask for an update (if 
the issue is actually the same).

Best Regards,
Strahil Nikolov


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] It appears that readdir is not cached for FUSE mounts

[Matthias Schniedermeyer]

On 10.02.20 16:21, Strahil Nikolov wrote:

On February 10, 2020 2:25:17 PM GMT+02:00, Matthias Schniedermeyer 
 wrote:

Hi


I would describe our basic use case for gluster as:
"data-store for a cold-standby application".

A specific application is installed on 2 hardware machines, the data is
kept in-sync between the 2 machines by a replica-2 gluster volume.
(IOW: "RAID 1")

At any one time only 1 machine has the volume mounted and the
application running. If the machine goes down the application is
started
on the remaining machine.
IOW at any one point in time there is only ever 1 "reader & writer"
running.

I profiled a performance problem we have with this application, which
unfortunately we can't modify.

The profile shows many "opendir/readdirp/releasedir" cycles, the
directory in question has about 1000 files and the application "stalls"
for several milliseconds any time it decides to do a readdir.
The volume is mounted via FUSE and it appears that said operation is
not
cached at all.

To provide a test-case i tried to replicate what the application does.
The problematic operation is nearly perfectly emulated just by using
"ls .".

I created a script that replicates how we use gluster and demonstrates
that a FUSE-mount appears to be lacking any caching of readdir.

A word about the test-environment:
2 identical servers
Dual Socket Xeon CPU E5-2640 v3 (8 cores, 2.60GHz, HT enabled)
RAM: 128GB DDR4 ECC (8x16GB)
Storage: 2TB Intel P3520 PCIe-NVMe-SSD
Network: Gluster: 10GB/s direct connect (no switch), external: 1Gbit/s
OS: CentOS 7.7, Installed with "Minimal" ISO, everything: Default
Up2Date as of: 2020-01-21 (Kernel: 3.10.0-1062.9.1.el7.x86_64)
SELinux: Disabled
SSH-Key for 1 -> 2 exchanged
Gluster 6.7 packages installed via 'centos-release-gluster6'

see attached: gluster-testcase-no-caching-of-dir-operations-for-fuse.sh

The meat of the testcase is this:
a profile of:
ls .
vs:
ls . . . . . . . . . .
(10 dots)


cat /root/profile-1-times | grep DIR | head -n 3

0.00   0.00 us   0.00 us   0.00 us  1
RELEASEDIR
0.27  66.79 us  66.79 us  66.79 us  1  OPENDIR
98.65   12190.30 us9390.88 us   14989.73 us  2
READDIRP


cat /root/profile-10-times | grep DIR | head -n 3

0.00   0.00 us   0.00 us   0.00 us 10
RELEASEDIR
0.64 108.02 us  85.72 us 131.96 us 10  OPENDIR
99.368388.64 us5174.71 us   14808.77 us 20
READDIRP

This testcase shows perfect scaling.
10 times the request, results in 10 times the gluster-operations.

I would say ideally there should be no difference in the number of
gluster-operations, regardless of how often a directory is read in a
short amount of time (with no changes in between)


Is there something we can do to enable caching or otherwise improve
performance?


Hi Matthias,

Have you tried the 'readdir-ahead' option .

> According to docs it is useful for ' improving sequential directory read 
performance' .
> I'm not sure how gluster defines sequential directory read, but it's worth 
trying.

readdir-ahead is enabled by default. Has been for several years.
In effect this option changes how many READDIRP OPs are executed for a
single "ls .".
(It takes more OPs when readdir-ahead is disabled.)


Also, you can try metadata caching , as described in:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/sect-directory_operations
The actual group should contain the following:
https://github.com/gluster/glusterfs/blob/master/extras/group-metadata-cache


Metadata-Caching, in general, works:
e.g. `stat FILE` is cached if executed repeatedly.

AFAICT the big exception to metadata-caching is readdir.



--
Matthias


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Permission denied at some directories/files after a split brain

[Alberto Bengoa]

Hello guys,

We are running GlusterFS 6.6 in Replicate mode (1 x 3). After a split-brain
and a massive heal process, we noticed that our app started to receive
thousands of permissions denied while trying to access files and
directories.

Exemple log of a failed access atempt to a specific directory:

[2020-02-10 10:38:17.402080] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-app_data-access-control:
client:
CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
gfid: 092f1e28-d6a8-4ca9-95d5-75dc8ad1c835,
req(uid:498,gid:498,perm:4,ngrps:1),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]
[2020-02-10 10:38:17.402182] E [MSGID: 115056]
[server-rpc-fops_v2.c:687:server4_opendir_cbk] 0-app_data-server: 6257941:
OPENDIR /mailboxes.old/8692/211411002/Old
(092f1e28-d6a8-4ca9-95d5-75dc8ad1c835), client:
CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
error-xlator: app_data-access-control [Permission denied]

The permission denied happens only to unprivileged users, even if that
unprivileged user is the directory owner. The root user is able to access
all files, and if we "touch" the file/directory as root it *sometimes*
fixes the problem.

We noticed inconsistent Access/Change dates. Here a stat of a directory
before touching it, showing these inconsistencies:

  File: ‘Old’
  Size: 4096   Blocks: 8  IO Block: 131072 directory
Device: 27h/39d Inode: 10388898073370567318  Links: 2
Access: (2775/drwxrwsr-x)  Uid: (  498/app)   Gid: (  498/app)
Access: 1970-01-01 01:00:00.0 +0100
Modify: 2020-02-07 13:21:10.365297527 +
Change: 1970-01-01 01:00:00.0 +0100
 Birth: -

I think this case is similar to the reported here[1] and discussed at
thread "ACL issue v6.6, v6.7, v7.1, v7.2", despite the fact that we are not
using libvirt. We do use ACLs, but not in this particular directory.

Any thoughts on this?

[1] - https://bugzilla.redhat.com/show_bug.cgi?id=1797099

Thanks,
Alberto Bengoa


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Samba Options

[Stefan Kania]

I have:
root@cluster-02:~# ls /var/lib/glusterd/groups
db-workload  gluster-block  metadata-cache  nl-cache  virt

and root@cluster-02:~# apt-file list glusterfs-server gives:
...
glusterfs-server: /var/lib/glusterd/groups/db-workload
glusterfs-server: /var/lib/glusterd/groups/gluster-block
glusterfs-server: /var/lib/glusterd/groups/metadata-cache
glusterfs-server: /var/lib/glusterd/groups/nl-cache
glusterfs-server: /var/lib/glusterd/groups/virt
...
So samba group :-(

And looking with "dpkg -l | grep glusterfs-server gives:
root@cluster-02:~# dpkg -l |grep glusterfs-server
ii  glusterfs-server  7.2-1 amd64 clustered file-system (server package)
seems to me the file is missing :-(


Am 10.02.20 um 09:53 schrieb Anoop C S:
> On Sun, 2020-02-09 at 15:44 +0100, Stefan Kania wrote:
>>
>> Am 08.02.20 um 11:33 schrieb Anoop C S:
>>> # gluster volume set  group samba
>> When i try to set the option I got the following error:
>> Unable to open file '/var/lib/glusterd/groups/samba'. Error: No such
>> file or directory
> 
> Do you have any other files under /var/lib/glusterd/groups/ ?
> 
>> And it's right there is no such file. I use Gluster 7.2-1 from the
>> offical repository for debian buster. Am I missing a package? I just
>> installeg glusterfs-server with it's dependencies
> 




smime.p7s
Description: S/MIME Cryptographic Signature


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] It appears that readdir is not cached for FUSE mounts

[Matthias Schniedermeyer]

Hi


I would describe our basic use case for gluster as:
"data-store for a cold-standby application".

A specific application is installed on 2 hardware machines, the data is
kept in-sync between the 2 machines by a replica-2 gluster volume.
(IOW: "RAID 1")

At any one time only 1 machine has the volume mounted and the
application running. If the machine goes down the application is started
on the remaining machine.
IOW at any one point in time there is only ever 1 "reader & writer"
running.

I profiled a performance problem we have with this application, which
unfortunately we can't modify.

The profile shows many "opendir/readdirp/releasedir" cycles, the
directory in question has about 1000 files and the application "stalls"
for several milliseconds any time it decides to do a readdir.
The volume is mounted via FUSE and it appears that said operation is not
cached at all.

To provide a test-case i tried to replicate what the application does.
The problematic operation is nearly perfectly emulated just by using
"ls .".

I created a script that replicates how we use gluster and demonstrates
that a FUSE-mount appears to be lacking any caching of readdir.

A word about the test-environment:
2 identical servers
Dual Socket Xeon CPU E5-2640 v3 (8 cores, 2.60GHz, HT enabled)
RAM: 128GB DDR4 ECC (8x16GB)
Storage: 2TB Intel P3520 PCIe-NVMe-SSD
Network: Gluster: 10GB/s direct connect (no switch), external: 1Gbit/s
OS: CentOS 7.7, Installed with "Minimal" ISO, everything: Default
Up2Date as of: 2020-01-21 (Kernel: 3.10.0-1062.9.1.el7.x86_64)
SELinux: Disabled
SSH-Key for 1 -> 2 exchanged
Gluster 6.7 packages installed via 'centos-release-gluster6'

see attached: gluster-testcase-no-caching-of-dir-operations-for-fuse.sh

The meat of the testcase is this:
a profile of:
ls .
vs:
ls . . . . . . . . . .
(10 dots)

> cat /root/profile-1-times | grep DIR | head -n 3
  0.00   0.00 us   0.00 us   0.00 us  1  RELEASEDIR
  0.27  66.79 us  66.79 us  66.79 us  1  OPENDIR
 98.65   12190.30 us9390.88 us   14989.73 us  2  READDIRP

> cat /root/profile-10-times | grep DIR | head -n 3
  0.00   0.00 us   0.00 us   0.00 us 10  RELEASEDIR
  0.64 108.02 us  85.72 us 131.96 us 10  OPENDIR
 99.368388.64 us5174.71 us   14808.77 us 20  READDIRP

This testcase shows perfect scaling.
10 times the request, results in 10 times the gluster-operations.

I would say ideally there should be no difference in the number of
gluster-operations, regardless of how often a directory is read in a
short amount of time (with no changes in between)


Is there something we can do to enable caching or otherwise improve
performance?



--
Matthias


gluster-testcase-no-caching-of-dir-operations-for-fuse.sh
Description: Bourne shell script


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Samba Options

[Anoop C S]

On Sun, 2020-02-09 at 15:44 +0100, Stefan Kania wrote:
> 
> Am 08.02.20 um 11:33 schrieb Anoop C S:
> > # gluster volume set  group samba
> When i try to set the option I got the following error:
> Unable to open file '/var/lib/glusterd/groups/samba'. Error: No such
> file or directory

Do you have any other files under /var/lib/glusterd/groups/ ?

> And it's right there is no such file. I use Gluster 7.2-1 from the
> offical repository for debian buster. Am I missing a package? I just
> installeg glusterfs-server with it's dependencies



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users