Re: [Gluster-users] Increase redundancy on existing disperse volume
I think I have replied all the questions you have asked. Let me know if you need any additional information. --- Ashish - Original Message - From: "Benjamin Kingston" To: "gluster-users" Sent: Tuesday, July 31, 2018 1:01:29 AM Subject: [Gluster-users] Increase redundancy on existing disperse volume I'm working to convert my 3x3 arbiter replicated volume into a disperse volume, however I have to work with the existing disks, maybe adding another 1 or 2 new disks if necessary. I'm hoping to destroy the bricks on one of the replicated nodes and build it into a I'm opting to host this volume on a set of controllers connected to a common backplane, I don't need help on this config, just on the constraints of the disperse volumes. I have some questions about the disperse functionality 1. If I create a 1 redundancy volume in the beginning, after I add more bricks, can I increase redundancy to 2 or 3 2. If I create the original volume with 6TB bricks, am I really stuck with 6TB bricks even if I add 2 or more 10TB bricks 3. Is it required to extend a volume by the same number or bricks it was created with? If the original volume is made with 3 bricks, do I have to always add capacity in 3 brick increments? -ben ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Community Meeting, 1 Aug
Quick update: Our next Gluster Community Meeting is August 1 at 15:00 UTC! Meeting agenda is https://bit.ly/gluster-community-meetings Anything you want to talk about? - amye -- Amye Scavarda | a...@redhat.com | Gluster Community Lead ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Old gluster PPA repositories
On Wed 1 Aug, 2018, 5:30 AM Igor Cicimov, wrote: > Hi, > > Was looking to upgrade some old 3.4 cluster running on Ubuntu but seems > older versions than 3.10 have been removed from the gluster PPA. How do > people proceed with upgrade in this case? From what I read in the upgrade > documentation I can upgrade from 3.4 to 3.6 but since the repository is > gone what are my options now? > > I tried to compile 3.6.9 on Ubuntu 14.04 but it errored out: > > make[5]: *** [keys.lo] Error 1 > make[4]: *** [all-recursive] Error 1 > make[3]: *** [all-recursive] Error 1 > make[2]: *** [all-recursive] Error 1 > make[1]: *** [all-recursive] Error 1 > make: *** [all] Error 2 > > By the way, why are the old PPA's being removed really? How are we to > automate when things are constantly disappearing? > Igor, Will check what happened and update here. Regards, Amar > Thanks, > Igor > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Old gluster PPA repositories
Hi, Was looking to upgrade some old 3.4 cluster running on Ubuntu but seems older versions than 3.10 have been removed from the gluster PPA. How do people proceed with upgrade in this case? From what I read in the upgrade documentation I can upgrade from 3.4 to 3.6 but since the repository is gone what are my options now? I tried to compile 3.6.9 on Ubuntu 14.04 but it errored out: make[5]: *** [keys.lo] Error 1 make[4]: *** [all-recursive] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 By the way, why are the old PPA's being removed really? How are we to automate when things are constantly disappearing? Thanks, Igor ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster Monthly Newsletter, July 2018
Gluster Monthly Newsletter, July 2018 mountpoint August 27-28 at the Vancouver Convention Center! We’ve published some schedule updates: http://mountpoint.io/ If you haven’t already registered, please do! Community Meeting - August 1, 15:00 UTC in #gluster-meeting on freenode. https://bit.ly/gluster-community-meetings has the agenda. Want swag for your meetup? https://www.gluster.org/events/ has a contact form for us to let us know about your Gluster meetup! We’d love to hear about Gluster presentations coming up, conference talks and gatherings. Let us know! Gluster Community Survey is just around the corner for August. Any questions you’d want to ask? Reply on the mailing list to this email. Contributors Top Contributing Companies: Red Hat, Comcast, DataLab, Gentoo Linux, Facebook, BioDec, Samsung, Etersoft Top Contributors in June: Amar Tumballi, Ravishankar N, Nigel Babu, Mohit Agrawal, Raghavendra G Noteworthy threads: [Gluster-users] Gluster Release cadence and version changes https://lists.gluster.org/pipermail/gluster-users/2018-July/034289.html [Gluster-users] Gluster Outreachy https://lists.gluster.org/pipermail/gluster-users/2018-July/034300.html [Gluster-users] Gluster Documentation Hackathon - 7/19 through 7/23 https://lists.gluster.org/pipermail/gluster-users/2018-July/034397.html [Gluster-users] Proposal to mark few features as Deprecated / Sunset from Version 5.0 https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html [Gluster-users] Feature Classification & Quality Expectation. https://lists.gluster.org/pipermail/gluster-users/2018-July/034412.html [Gluster-users] Subject: Help needed in improving monitoring in Gluster https://lists.gluster.org/pipermail/gluster-users/2018-July/034435.html [Gluster-devel] Re-thinking gluster regression logging https://lists.gluster.org/pipermail/gluster-devel/2018-July/054946.html [Gluster-devel] [Gluster-infra] Switching mailman to https only https://lists.gluster.org/pipermail/gluster-devel/2018-July/054947.html [Gluster-devel] Gluster project's intentions to move to python3 as default. https://lists.gluster.org/pipermail/gluster-devel/2018-July/054975.html [Gluster-devel] Release 5: Release targeted for first week of October-2018 https://lists.gluster.org/pipermail/gluster-devel/2018-July/054981.html [Gluster-devel] Release 5: Master branch health report (Week of 23rd July) https://lists.gluster.org/pipermail/gluster-devel/2018-July/055028.html [Gluster-devel] Release 5: Master branch health report (Week of 30th July) https://lists.gluster.org/pipermail/gluster-devel/2018-July/055077.html [Gluster-devel] Tests failing for distributed regression framework https://lists.gluster.org/pipermail/gluster-devel/2018-July/054982.html [Gluster-devel] regression failures on afr/split-brain-resolution https://lists.gluster.org/pipermail/gluster-devel/2018-July/055018.html [Gluster-devel] How long should metrics collection on a cluster take? https://lists.gluster.org/pipermail/gluster-devel/2018-July/055024.html [Gluster-devel] Failing multiplex regressions! https://lists.gluster.org/pipermail/gluster-devel/2018-July/055026.html [Gluster-devel] Maintainer's Meeting on 23rd July, 2018: Meeting minutes https://lists.gluster.org/pipermail/gluster-devel/2018-July/055056.html [Gluster-devel] Wireshark dissectors for Gluster 4.0 https://lists.gluster.org/pipermail/gluster-devel/2018-July/055060.html [Gluster-devel] FreeBSD smoke test may fail for older changes, rebase needed https://lists.gluster.org/pipermail/gluster-devel/2018-July/055070.html [Gluster-infra] Postmortem for Jenkins Outage on 20/07/18 https://lists.gluster.org/pipermail/gluster-infra/2018-July/004654.html [Gluster-infra] [Gluster-devel] Github teams/repo cleanup https://lists.gluster.org/pipermail/gluster-infra/2018-July/004667.html [Gluster-infra] Gerrit downtime on Aug 8, 2016 https://lists.gluster.org/pipermail/gluster-infra/2018-July/004694.html Events: Mountpoint, Aug 27-28 in Vancouver, BC Open Source Summit North America, Aug 29-31 in Vancouver, BC Open Source Summit Europe, Oct 22-24 in Edinburgh, UK KubeCon North America 2018, Dec 11-13, in Seattle, US FOSDEM, Feb 2-3 2019 in Brussels, Belgium -- Amye Scavarda | a...@redhat.com | Gluster Community Lead ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Increase redundancy on existing disperse volume
I'm working to convert my 3x3 arbiter replicated volume into a disperse volume, however I have to work with the existing disks, maybe adding another 1 or 2 new disks if necessary. I'm hoping to destroy the bricks on one of the replicated nodes and build it into a I'm opting to host this volume on a set of controllers connected to a common backplane, I don't need help on this config, just on the constraints of the disperse volumes. I have some questions about the disperse functionality 1. If I create a 1 redundancy volume in the beginning, after I add more bricks, can I increase redundancy to 2 or 3 2. If I create the original volume with 6TB bricks, am I really stuck with 6TB bricks even if I add 2 or more 10TB bricks 3. Is it required to extend a volume by the same number or bricks it was created with? If the original volume is made with 3 bricks, do I have to always add capacity in 3 brick increments? -ben ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Need advice about optimal configuration for VM hosting.
Hi everyone. I need some sage advice for upcoming upgrades we're planning for our Gluster array. I'll start by describing our server cluster: We currently have 3 Proxmox nodes. Two of them are the workhorses, running 12 of our production VMs and a handful of dev VMs that don't see the heavy use of the production VMs. The third one is a slower machine that mostly acts to tell Proxmox how to avoid split-brain issues, but is also running the third Galera node for our database. Galera is actually running on the local disks of each Proxmox node because of the performance issues we've been having with the Gluster array. Under that, we have our Gluster array, which currently consists of two nodes in a Replicated volume type. Each node has two bricks: one for VMs and one for our mail. The problem is that it's a bit overloaded, mostly thanks the the VM traffic. I've got a couple new nodes to throw into that array, but in the past, adding a third node killed performance so badly that even after the array rebuild, we were looking at a decrease in disk performance of about 50%. I'm thinking that adding a fourth node might improve things, but that still means waiting for the Gluster array to rebuild, suffering the performance degradation of the third node, then adding the fourth, suffering the performance degradation of the rebuild again, and then... what? See what happens and add more nodes to hopefully further improve things? That doesn't sound like a good strategy to me. There are too many questions, and too much hoping things get better. Another strategy I was thinking of, might be to build a whole new Gluster array with the latest version of Gluster. Then, instead of suffering the long array rebuild times, we can move VMs one at a time to the new Gluster disk. It's more work for me, but probably less pain for our customers. I was thinking that the new array should be 4 nodes in a Replicated-Distributed volume type, and after we've moved our data off the old array, nuking the old nodes and adding them to the new array. My understanding is that we'd have to add new nodes two at a time to ensure the replication makes sense to Gluster. ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Rebalance taking > 2 months
Is it possible to pause the rebalance to get those number? I'm hesitant to stop the rebalance and have to redo the entire thing again. On Tue, Jul 31, 2018 at 11:40 AM, Nithya Balachandran wrote: > > > On 31 July 2018 at 19:44, Rusty Bower wrote: > >> I'll figure out what hasn't been rebalanced yet and run the script. >> >> There's only a single client accessing this gluster volume, and while the >> rebalance is taking place, the I am only able to read/write to the volume >> at around 3MB/s. If I log onto one of the bricks, I can read/write to the >> physical volumes at speed greater than 100MB/s (which is what I would >> expect). >> > > What are the numbers when accessing the volume when rebalance is not > running? > Regards, > Nithya > >> >> Thanks! >> Rusty >> >> On Tue, Jul 31, 2018 at 3:28 AM, Nithya Balachandran > > wrote: >> >>> Hi Rusty, >>> >>> A rebalance involves 2 steps: >>> >>>1. Setting a new layout on a directory >>>2. Migrating any files inside that directory that hash to a >>>different subvol based on the new layout set in step 1. >>> >>> >>> A few things to keep in mind : >>> >>>- Any new content created on this volume will currently go to the >>>newly added brick. >>>- Having a more equitable file distribution is beneficial but you >>>might not need to do a complete rebalance to do this. You can run the >>>script on just enough directories to free up space on your older bricks. >>>This should be done on bricks which contains large files to speed this >>> up. >>> >>> Do the following on one of the server nodes: >>> >>>- Create a tmp mount point and mount the volume using the rebalance >>>volfile >>>- mkdir /mnt/rebal >>> - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal >>>- Select a directory in the volume which contains a lot of large >>>files and which has not been processed by the rebalance yet - the lower >>>down in the tree the better. Check the rebalance logs to figure out which >>>dirs have not been processed yet. >>> - cd /mnt/rebal/ >>> - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10 >>> bash process_dir.sh;done >>>- You can run this for different values of and on >>>multiple server nodes in parallel as long as the directory trees for the >>>different don't overlap. >>>- Do this for multiple directories until the disk space used reduces >>>on the older bricks. >>> >>> This is a very simple script. Let me know how it works - we can always >>> tweak it for your particular data set. >>> >>> >>> >and performance is basically garbage while it rebalances >>> Can you provide more detail on this? What kind of effects are you seeing? >>> How many clients access this volume? >>> >>> >>> Regards, >>> Nithya >>> >>> On 30 July 2018 at 22:18, Nithya Balachandran >>> wrote: >>> I have not documented this yet - I will send you the steps tomorrow. Regards, Nithya On 30 July 2018 at 20:23, Rusty Bower wrote: > That would be awesome. Where can I find these? > > Rusty > > Sent from my iPhone > > On Jul 30, 2018, at 03:40, Nithya Balachandran > wrote: > > Hi Rusty, > > Sorry for the delay getting back to you. I had a quick look at the > rebalance logs - it looks like the estimates are based on the time taken > to > rebalance the smaller files. > > We do have a scripting option where we can use virtual xattrs to > trigger file migration from a mount point. That would speed things up. > > > Regards, > Nithya > > On 28 July 2018 at 07:11, Rusty Bower wrote: > >> Just wanted to ping this to see if you guys had any thoughts, or >> other scripts I can run for this stuff. It's still predicting another 90 >> days to rebalance this, and performance is basically garbage while it >> rebalances. >> >> Rusty >> >> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower >> wrote: >> >>> datanode03 is the newest brick >>> >>> the bricks had gotten pretty full, which I think might be part of >>> the issue: >>> - datanode01 /dev/sda1 51T 48T 3.3T 94% /mnt/data >>> - datanode02 /dev/sda1 51T 48T 3.4T 94% /mnt/data >>> - datanode03 /dev/md0 128T 4.6T 123T 4% /mnt/data >>> >>> each of the bricks are on a completely separate disk from the OS >>> >>> I'll shoot you the log files offline :) >>> >>> Thanks! >>> Rusty >>> >>> On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran < >>> nbala...@redhat.com> wrote: >>> Hi Rusty, Sorry I took so long to get back to you. Which is the newly added brick? I see datanode02 has not picked up any files for migration which is odd. How full are the individual bricks (df -h ) output.
Re: [Gluster-users] Rebalance taking > 2 months
On 31 July 2018 at 19:44, Rusty Bower wrote: > I'll figure out what hasn't been rebalanced yet and run the script. > > There's only a single client accessing this gluster volume, and while the > rebalance is taking place, the I am only able to read/write to the volume > at around 3MB/s. If I log onto one of the bricks, I can read/write to the > physical volumes at speed greater than 100MB/s (which is what I would > expect). > What are the numbers when accessing the volume when rebalance is not running? Regards, Nithya > > Thanks! > Rusty > > On Tue, Jul 31, 2018 at 3:28 AM, Nithya Balachandran > wrote: > >> Hi Rusty, >> >> A rebalance involves 2 steps: >> >>1. Setting a new layout on a directory >>2. Migrating any files inside that directory that hash to a different >>subvol based on the new layout set in step 1. >> >> >> A few things to keep in mind : >> >>- Any new content created on this volume will currently go to the >>newly added brick. >>- Having a more equitable file distribution is beneficial but you >>might not need to do a complete rebalance to do this. You can run the >>script on just enough directories to free up space on your older bricks. >>This should be done on bricks which contains large files to speed this up. >> >> Do the following on one of the server nodes: >> >>- Create a tmp mount point and mount the volume using the rebalance >>volfile >>- mkdir /mnt/rebal >> - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal >>- Select a directory in the volume which contains a lot of large >>files and which has not been processed by the rebalance yet - the lower >>down in the tree the better. Check the rebalance logs to figure out which >>dirs have not been processed yet. >> - cd /mnt/rebal/ >> - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10 >> bash process_dir.sh;done >>- You can run this for different values of and on >>multiple server nodes in parallel as long as the directory trees for the >>different don't overlap. >>- Do this for multiple directories until the disk space used reduces >>on the older bricks. >> >> This is a very simple script. Let me know how it works - we can always >> tweak it for your particular data set. >> >> >> >and performance is basically garbage while it rebalances >> Can you provide more detail on this? What kind of effects are you seeing? >> How many clients access this volume? >> >> >> Regards, >> Nithya >> >> On 30 July 2018 at 22:18, Nithya Balachandran >> wrote: >> >>> I have not documented this yet - I will send you the steps tomorrow. >>> >>> Regards, >>> Nithya >>> >>> On 30 July 2018 at 20:23, Rusty Bower wrote: >>> That would be awesome. Where can I find these? Rusty Sent from my iPhone On Jul 30, 2018, at 03:40, Nithya Balachandran wrote: Hi Rusty, Sorry for the delay getting back to you. I had a quick look at the rebalance logs - it looks like the estimates are based on the time taken to rebalance the smaller files. We do have a scripting option where we can use virtual xattrs to trigger file migration from a mount point. That would speed things up. Regards, Nithya On 28 July 2018 at 07:11, Rusty Bower wrote: > Just wanted to ping this to see if you guys had any thoughts, or other > scripts I can run for this stuff. It's still predicting another 90 days to > rebalance this, and performance is basically garbage while it rebalances. > > Rusty > > On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower > wrote: > >> datanode03 is the newest brick >> >> the bricks had gotten pretty full, which I think might be part of the >> issue: >> - datanode01 /dev/sda1 51T 48T 3.3T 94% /mnt/data >> - datanode02 /dev/sda1 51T 48T 3.4T 94% /mnt/data >> - datanode03 /dev/md0 128T 4.6T 123T 4% /mnt/data >> >> each of the bricks are on a completely separate disk from the OS >> >> I'll shoot you the log files offline :) >> >> Thanks! >> Rusty >> >> On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran < >> nbala...@redhat.com> wrote: >> >>> Hi Rusty, >>> >>> Sorry I took so long to get back to you. >>> >>> Which is the newly added brick? I see datanode02 has not picked up >>> any files for migration which is odd. >>> How full are the individual bricks (df -h ) output. >>> Is each of your bricks in a separate partition? >>> Can you send me the rebalance logs from all 3 nodes (offline if you >>> prefer)? >>> >>> We can try using scripts to speed up the rebalance if you prefer. >>> >>> Regards, >>> Nithya >>> >>> >>> >>> On 16 July 2018 at 22:06, Rusty Bower wrote: >>>
Re: [Gluster-users] Rebalance taking > 2 months
I'll figure out what hasn't been rebalanced yet and run the script. There's only a single client accessing this gluster volume, and while the rebalance is taking place, the I am only able to read/write to the volume at around 3MB/s. If I log onto one of the bricks, I can read/write to the physical volumes at speed greater than 100MB/s (which is what I would expect). Thanks! Rusty On Tue, Jul 31, 2018 at 3:28 AM, Nithya Balachandran wrote: > Hi Rusty, > > A rebalance involves 2 steps: > >1. Setting a new layout on a directory >2. Migrating any files inside that directory that hash to a different >subvol based on the new layout set in step 1. > > > A few things to keep in mind : > >- Any new content created on this volume will currently go to the >newly added brick. >- Having a more equitable file distribution is beneficial but you >might not need to do a complete rebalance to do this. You can run the >script on just enough directories to free up space on your older bricks. >This should be done on bricks which contains large files to speed this up. > > Do the following on one of the server nodes: > >- Create a tmp mount point and mount the volume using the rebalance >volfile >- mkdir /mnt/rebal > - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal >- Select a directory in the volume which contains a lot of large files >and which has not been processed by the rebalance yet - the lower down in >the tree the better. Check the rebalance logs to figure out which dirs have >not been processed yet. > - cd /mnt/rebal/ > - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10 bash > process_dir.sh;done >- You can run this for different values of and on >multiple server nodes in parallel as long as the directory trees for the >different don't overlap. >- Do this for multiple directories until the disk space used reduces >on the older bricks. > > This is a very simple script. Let me know how it works - we can always > tweak it for your particular data set. > > > >and performance is basically garbage while it rebalances > Can you provide more detail on this? What kind of effects are you seeing? > How many clients access this volume? > > > Regards, > Nithya > > On 30 July 2018 at 22:18, Nithya Balachandran wrote: > >> I have not documented this yet - I will send you the steps tomorrow. >> >> Regards, >> Nithya >> >> On 30 July 2018 at 20:23, Rusty Bower wrote: >> >>> That would be awesome. Where can I find these? >>> >>> Rusty >>> >>> Sent from my iPhone >>> >>> On Jul 30, 2018, at 03:40, Nithya Balachandran >>> wrote: >>> >>> Hi Rusty, >>> >>> Sorry for the delay getting back to you. I had a quick look at the >>> rebalance logs - it looks like the estimates are based on the time taken to >>> rebalance the smaller files. >>> >>> We do have a scripting option where we can use virtual xattrs to trigger >>> file migration from a mount point. That would speed things up. >>> >>> >>> Regards, >>> Nithya >>> >>> On 28 July 2018 at 07:11, Rusty Bower wrote: >>> Just wanted to ping this to see if you guys had any thoughts, or other scripts I can run for this stuff. It's still predicting another 90 days to rebalance this, and performance is basically garbage while it rebalances. Rusty On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower wrote: > datanode03 is the newest brick > > the bricks had gotten pretty full, which I think might be part of the > issue: > - datanode01 /dev/sda1 51T 48T 3.3T 94% /mnt/data > - datanode02 /dev/sda1 51T 48T 3.4T 94% /mnt/data > - datanode03 /dev/md0 128T 4.6T 123T 4% /mnt/data > > each of the bricks are on a completely separate disk from the OS > > I'll shoot you the log files offline :) > > Thanks! > Rusty > > On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran < > nbala...@redhat.com> wrote: > >> Hi Rusty, >> >> Sorry I took so long to get back to you. >> >> Which is the newly added brick? I see datanode02 has not picked up >> any files for migration which is odd. >> How full are the individual bricks (df -h ) output. >> Is each of your bricks in a separate partition? >> Can you send me the rebalance logs from all 3 nodes (offline if you >> prefer)? >> >> We can try using scripts to speed up the rebalance if you prefer. >> >> Regards, >> Nithya >> >> >> >> On 16 July 2018 at 22:06, Rusty Bower wrote: >> >>> Thanks for the reply Nithya. >>> >>> 1. glusterfs 4.1.1 >>> >>> 2. Volume Name: data >>> Type: Distribute >>> Volume ID: 294d95ce-0ff3-4df9-bd8c-a52fc50442ba >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 3 >>> Transport-type: tcp >>> Bricks:
Re: [Gluster-users] Rebalance taking > 2 months
Hi Rusty, A rebalance involves 2 steps: 1. Setting a new layout on a directory 2. Migrating any files inside that directory that hash to a different subvol based on the new layout set in step 1. A few things to keep in mind : - Any new content created on this volume will currently go to the newly added brick. - Having a more equitable file distribution is beneficial but you might not need to do a complete rebalance to do this. You can run the script on just enough directories to free up space on your older bricks. This should be done on bricks which contains large files to speed this up. Do the following on one of the server nodes: - Create a tmp mount point and mount the volume using the rebalance volfile - mkdir /mnt/rebal - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal - Select a directory in the volume which contains a lot of large files and which has not been processed by the rebalance yet - the lower down in the tree the better. Check the rebalance logs to figure out which dirs have not been processed yet. - cd /mnt/rebal/ - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10 bash process_dir.sh;done - You can run this for different values of and on multiple server nodes in parallel as long as the directory trees for the different don't overlap. - Do this for multiple directories until the disk space used reduces on the older bricks. This is a very simple script. Let me know how it works - we can always tweak it for your particular data set. >and performance is basically garbage while it rebalances Can you provide more detail on this? What kind of effects are you seeing? How many clients access this volume? Regards, Nithya On 30 July 2018 at 22:18, Nithya Balachandran wrote: > I have not documented this yet - I will send you the steps tomorrow. > > Regards, > Nithya > > On 30 July 2018 at 20:23, Rusty Bower wrote: > >> That would be awesome. Where can I find these? >> >> Rusty >> >> Sent from my iPhone >> >> On Jul 30, 2018, at 03:40, Nithya Balachandran >> wrote: >> >> Hi Rusty, >> >> Sorry for the delay getting back to you. I had a quick look at the >> rebalance logs - it looks like the estimates are based on the time taken to >> rebalance the smaller files. >> >> We do have a scripting option where we can use virtual xattrs to trigger >> file migration from a mount point. That would speed things up. >> >> >> Regards, >> Nithya >> >> On 28 July 2018 at 07:11, Rusty Bower wrote: >> >>> Just wanted to ping this to see if you guys had any thoughts, or other >>> scripts I can run for this stuff. It's still predicting another 90 days to >>> rebalance this, and performance is basically garbage while it rebalances. >>> >>> Rusty >>> >>> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower >>> wrote: >>> datanode03 is the newest brick the bricks had gotten pretty full, which I think might be part of the issue: - datanode01 /dev/sda1 51T 48T 3.3T 94% /mnt/data - datanode02 /dev/sda1 51T 48T 3.4T 94% /mnt/data - datanode03 /dev/md0 128T 4.6T 123T 4% /mnt/data each of the bricks are on a completely separate disk from the OS I'll shoot you the log files offline :) Thanks! Rusty On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran < nbala...@redhat.com> wrote: > Hi Rusty, > > Sorry I took so long to get back to you. > > Which is the newly added brick? I see datanode02 has not picked up > any files for migration which is odd. > How full are the individual bricks (df -h ) output. > Is each of your bricks in a separate partition? > Can you send me the rebalance logs from all 3 nodes (offline if you > prefer)? > > We can try using scripts to speed up the rebalance if you prefer. > > Regards, > Nithya > > > > On 16 July 2018 at 22:06, Rusty Bower wrote: > >> Thanks for the reply Nithya. >> >> 1. glusterfs 4.1.1 >> >> 2. Volume Name: data >> Type: Distribute >> Volume ID: 294d95ce-0ff3-4df9-bd8c-a52fc50442ba >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 3 >> Transport-type: tcp >> Bricks: >> Brick1: datanode01:/mnt/data/bricks/data >> Brick2: datanode02:/mnt/data/bricks/data >> Brick3: datanode03:/mnt/data/bricks/data >> Options Reconfigured: >> performance.readdir-ahead: on >> >> 3. >> Node Rebalanced-files >> size scanned failures skipped status run >> time in h:m:s >>- --- >> --- --- --- --- >> -- >>localhost36822
