Re: [Gluster-users] [Gluster-devel] A question of GlusterFS dentries!
On Wed, Nov 2, 2016 at 9:38 AM, Raghavendra Gowdappa wrote: > > > - Original Message - > > From: "Keiviw" > > To: gluster-de...@gluster.org > > Sent: Tuesday, November 1, 2016 12:41:02 PM > > Subject: [Gluster-devel] A question of GlusterFS dentries! > > > > Hi, > > In GlusterFS distributed volumes, listing a non-empty directory was slow. > > Then I read the dht codes and found the reasons. But I was confused that > > GlusterFS dht travesed all the bricks(in the volume) sequentially,why not > > use multi-thread to read dentries from multiple bricks simultaneously. > > That's a question that's always puzzled me, Couly you please tell me > > something about this??? > > readdir across subvols is sequential mostly because we have to support > rewinddir(3). Sorry. seekdir(3) is the more relevant function here. Since rewinddir resets the dir stream to beginning, its not much of a difficulty to support rewinddir with parallel readdirs across subvols. > We need to maintain the mapping of offset and dentry across multiple > invocations of readdir. In other words if someone did a rewinddir to an > offset corresponding to earlier dentry, subsequent readdirs should return > same set of dentries what the earlier invocation of readdir returned. For > example, in an hypothetical scenario, readdir returned following dentries: > > 1. a, off=10 > 2. b, off=2 > 3. c, off=5 > 4. d, off=15 > 5. e, off=17 > 6. f, off=13 > > Now if we did rewinddir to off 5 and issue readdir again we should get > following dentries: > (c, off=5), (d, off=15), (e, off=17), (f, off=13) > > Within a subvol backend filesystem provides rewinddir guarantee for the > dentries present on that subvol. However, across subvols it is the > responsibility of DHT to provide the above guarantee. Which means we > should've some well defined order in which we send readdir calls (Note that > order is not well defined if we do a parallel readdir across all subvols). > So, DHT has sequential readdir which is a well defined order of reading > dentries. > > To give an example if we have another subvol - subvol2 - (in addiction to > the subvol above - say subvol1) with following listing: > 1. g, off=16 > 2. h, off=20 > 3. i, off=3 > 4. j, off=19 > > With parallel readdir we can have many ordering like - (a, b, g, h, i, c, > d, e, f, j), (g, h, a, b, c, i, j, d, e, f) etc. Now if we do (with readdir > done parallely): > > 1. A complete listing of the directory (which can be any one of 10P1 = 10 > ways - I hope math is correct here). > 2. Do rewinddir (20) > > We cannot predict what are the set of dentries that come _after_ offset > 20. However, if we do a readdir sequentially across subvols there is only > one directory listing i.e, (a, b, c, d, e, f, g, h, i, j). So, its easier > to support rewinddir. > > If there is no POSIX requirement for rewinddir support, I think a parallel > readdir can easily be implemented (which improves performance too). But > unfortunately rewinddir is still a POSIX requirement. This also opens up > another possibility of a "no-rewinddir-support" option in DHT, which if > enabled results in parallel readdirs across subvols. What I am not sure is > how many users still use rewinddir? If there is a critical mass which wants > performance with a tradeoff of no rewinddir support this can be a good > feature. > > +gluster-users to get an opinion on this. > > regards, > Raghavendra > > > > > > > > > > > > > > > ___ > > Gluster-devel mailing list > > gluster-de...@gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-devel > ___ > Gluster-devel mailing list > gluster-de...@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > -- Raghavendra G ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] 3.8.5 gfapi problem
Hello! We got problems after upgrade drom 3.8.4 to 3.8.5, namely we got block I/O error in device 'drive-ide0-0-0': Input/output error (5) Really qemu complains about no boot device. I created bug report for it https://bugzilla.redhat.com/show_bug.cgi?id=1390521 but there is no attention to it. Downgrading back to 3.8.4 solved problem, but Thank you! ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Weekly community meeting - Nov 02, 2016
Hi all, This week we discussed 3 topics (again!). 1. The FSFE etherpad service (where we host the meeting agenda and a lot other gluster docs) will be shutdown, so we need to find an alternative. We decided to make use of etherpads as temporary tools of live collaboration, and using github wikis to permanently record the document. Our first task is to identify and record known/live etherpads. Nigel and I will start working on this, with our first target being the community meeting agenda. 2. Nigel has created a centos-ci job for Glusto, which is nearly ready for use. 3. We continued discussion on 'Recognizing contributors', and we couldn't arrive at a conclusion again. This discussion will continue on the mailing lists. I've pasted the meeting minutes below for easy reference. This will also be available from a github wiki soon. I'll be hosting next weeks meeting. See you all next week, same time & same place. ~kaushal ## Logs - Minutes: https://meetbot.fedoraproject.org/gluster-meeting/2016-11-02/weekly_community_meeting_nov_2,_2016.2016-11-02-12.00.html - Minutes (text): https://meetbot.fedoraproject.org/gluster-meeting/2016-11-02/weekly_community_meeting_nov_2,_2016.2016-11-02-12.00.txt - Log: https://meetbot.fedoraproject.org/gluster-meeting/2016-11-02/weekly_community_meeting_nov_2,_2016.2016-11-02-12.00.log.html ## Topics of Discussion ### Host for next week (DON'T FORGET) - Next weeks meeting moderator: kshlm - kkeithley will host the Nov 16 meeting ### Open Floor - Recognising community contributors(Manikandan Selvaganesh) - https://glusterfs.biterg.io/app/kibana#/dashboard/Overview - [nigelb] Are we clear on what is being asked for? What is the problem we are trying to solve - People want to be rewarded for their contributions - As a reward for their work make them maintainers - Need to measure quality, not quantity. - Super tough - nigelb will be taking discussion to mailing list - Sending up EasyFix bugs(assigned with a owner) on mailing list so that new contributors can get started(May be?) (Manikandan Selvaganesh) - Skipped again - Glusto Job on Centos CI - https://ci.centos.org/job/gluster_glusto - Job setup, needs tweaking to get it 100% functional. - Leverage learaning from 3.9 test day into some tests. - FSFE pad service is about to be decomissioned. Need to find an alternative. - https://wiki.fsfe.org/Teams/System-Hackers/Decommissioning-Pad - Pads will turn read-only on Dec 1. And will be deleted Feb 1. - Options - beta.etherpad.org - https://pad.riseup.net/ - https://github.com/ether/etherpad-lite/wiki/Sites-that-run-Etherpad-Lite - Etherpad for live editing documents + Github wiki for permanent records. - First find and preserve existing pads. Pick alternative later. ## Updates ### GlusterFS 4.0 - GD2 (kshlm) - Volgen-2.0 design and prototype started. More info at https://www.gluster.org/pipermail/gluster-devel/2016-October/051297.html - Multiplexing - https://www.gluster.org/pipermail/gluster-devel/2016-November/051364.html ### GlusterFS 3.9 - _None_ ### GlusterFS 3.8 - _None_ ### GlusterFS 3.7 - Release planned for later today. Got delayed because of holidays in India. ### GlusterFS 3.6 - _None_ ### Other initiatives and related projects Infrastructure - Ongoing work on getting Glusto jobs on Centos CI - Network issue between Centos CI and review.gluster.org NFS Ganesha - ganesha 2.4.1 and (lib)ntirpc-1.4.3) were released. packages in Fedora Updates-Testing (soon Updates) and Ubuntu Launchpad. Other distributions soon. - now working on 2.5, emphasis on performance and memory consum Samba - Regression caused due to https://review.gluster.org/#/c/15332/ is being tracked by upstream bug https://bugzilla.samba.org/show_bug.cgi?id=12404 so that the fix from master gets backported to release branches. Heketi/Containers - We have a new repo under Gluster org in Github for anything and everything related to Gluster as storage in containers. Refer to https://github.com/gluster/container-storage Testing/Glusto - https://ci.centos.org/job/gluster_glusto ### Last weeks AIs - rastar_afk/ndevos/jdarcy to improve cleanup to control the processes that test starts. - _No updates_ - atinm to poke 3.9 release mgrs to finish and release - _No updates_ - obnox to starting discussion of Samba memory solutions - _No updates_ - jdarcy to discuss dbench smoke test failures on email - _No updates_ ### Other Updates - amye sent out the monthly newsletter - https://www.gluster.org/pipermail/gluster-users/2016-November/028920.html ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Shared Heal Times
On 2/11/2016 7:24 PM, David Gossage wrote: What are final settings you ended up with after reverting? Volume Name: datastore4 Type: Replicate Volume ID: 0ba131ef-311d-4bb1-be46-596e83b2f6ce Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vnb.proxmox.softlog:/tank/vmdata/datastore4 Brick2: vng.proxmox.softlog:/tank/vmdata/datastore4 Brick3: vna.proxmox.softlog:/tank/vmdata/datastore4 Options Reconfigured: performance.readdir-ahead: on cluster.data-self-heal: on features.shard: on cluster.quorum-type: auto cluster.server-quorum-type: server nfs.disable: on nfs.addr-namelookup: off nfs.enable-ino32: off performance.strict-write-ordering: off performance.stat-prefetch: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off cluster.eager-lock: enable network.remote-dio: enable features.shard-block-size: 64MB cluster.granular-entry-heal: yes cluster.locking-scheme: granular -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Shared Heal Times
On Tue, Nov 1, 2016 at 9:59 PM, Lindsay Mathieson < lindsay.mathie...@gmail.com> wrote: > Just an update - after resetting all the heal "optimisations" :) I set, in > general heals are much faster and back to normal. I've done several rolling > upgrades with the servers since, rebooting each one in turn. Usually around > 300 64MB shards will need healing after each boot. Its spends about 2-3 > minutes doing some fairly intensive CPU, then another 10 minutes to > complete the heal. All up around 15 minutes per server. I'm more than > satisfied with that. > > > So no real problem other than PEBKAC. > > > Moral of the story - as always, tuning settings for optimisation almost > never works. > What are final settings you ended up with after reverting? > > > > thanks, > > -- > Lindsay Mathieson > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users