Re: [Gluster-users] Extremely slow file listing in folders with many files

2020-05-20 Thread Susant Palai 
Artem, could you create a github with all the details. Easier to track
there and collaborate. Please update a snippet of the log file there as
well.

On Tue, May 19, 2020 at 11:30 PM Artem Russakovskii 
wrote:

> Hi Susant,
>
> As I just mentioned in the other thread, this happens with heal queue at 0
> across all of our 1x4 replicate gluster instances, and without any downtime
> of any of the nodes. For example, I just reran ls in the same dir I emailed
> about yesterday, and it's once again full of holes=1.
>
> I haven't added a new brick to the cluster in years - it's been the same 4
> bricks.
>
> We need to get to the bottom of this.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | @ArtemR 
>
>
> On Mon, May 18, 2020 at 11:40 PM Susant Palai  wrote:
>
>> From the logs it looks like, most of the directories needing heal and
>> this could slow down the ls -R operation. Possible reason for holes=1 in
>> the message could be that one of the brick was down when mkdir was going on
>> or you might have added a new brick recently to the cluster.
>>
>> On Tue, May 19, 2020 at 3:56 AM Artem Russakovskii 
>> wrote:
>>
>>> I've launched an ls in a folder with ~100k files/dirs 20 minutes ago.
>>> It's still not done and the log is filled with tens of thousands of these.
>>> This makes gluster almost unusable right now.
>>>
>>> [2020-05-18 22:23:35.919504] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 5c539141-dbe5-484a-acc0-acc369d2e55b). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919526] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 686ed221-0b5a-4412-bd70-385b707cc63e). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919534] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 04459d23-f77d-4ad4-baf6-bb5c39dc59b6). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919545] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = e0beed96-4f2b-4e4b-8e02-963a159da4ff). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919552] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 4393a4a8-3768-4acf-a4bd-33152f79c7b9). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919564] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 2ec6e22b-2226-482f-9149-4ef9259f2777). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919571] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 6047055d-59ec-405b-ab9d-69cf3112d51b). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919585] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 6cc7de36-ddd6-4daf-b219-3d823b5020d1). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:35.919592] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 27dd2dfa-f8f2-48e5-b59d-a407293a89f7). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286582] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = b4b8fcde-c0d0-4d4e-ae49-c09b4a4f5360). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286631] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 80d7664c-aa3d-4675-bea4-389f3bef9d08). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286641] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 6b5a6b00-1c77-4c2a-b92a-000f00b03b8c). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286649] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = b6be91a5-c7ff-441f-b495-3aaa360c66aa). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286663] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 1d22681c-f55c-4021-82ee-86f1a91caa32). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286671] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 93d3cecf-b340-46a7-8fcb-7139d6337021). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286713] I [MSGID: 109063]
>>> [dht-layout.c:659:dht_layout_normalize] 0-apkmirror_data1-dht: Found
>>> anomalies in (null) (gfid = 93dfb66d-c60c-4708-826b-a1078a85c741). Holes=1
>>> overlaps=0
>>> [2020-05-18 22:23:36.286722

[Gluster-users] [Gluster-devel] Announcing Gluster release 7.6

2020-05-20 Thread Rinku Kothiya 
Hi,

The Gluster community is pleased to announce the release of Gluster7.6
(packages available at [1]).
Release notes for the release can be found at [2].

Major changes, features and limitations addressed in this release:
None

Please Note: Some of the packages are unavailable and we are working on it.
We will release them soon.

Thanks,
Gluster community

References:

[1] Packages for 7.6:
https://download.gluster.org/pub/gluster/glusterfs/7/7.6/

[2] Release notes for 7.6:
https://docs.gluster.org/en/latest/release-notes/7.6/




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users