Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread valdis . kletnieks
On Wed, 01 Nov 2017 15:54:04 -0700, John Hanks said: > illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device Check 'df -i' to make sure no file systems are out of inodes. That's ___ gpfsug-discuss mailing list gpfsug-discuss at spectrum

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Yaron Daniel
Hi Please check mmdf output to see that MetaData disks are not full, or you have i-nodes issue. In case you have Independent File-Sets , please run : mmlsfileset -L -i to get the status of each fileset inodes. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Serv

[gpfsug-discuss] Antwort: [Newsletter] Re: Combine different rules - tip: use mmfind & co; FOR FILESET; FILESET_NAME

2017-11-02 Thread Matthias . Knigge
Thanks for this tip. I will try these commands and give feedback in the next week. Matthias Von:"Marc A Kaplan" An: gpfsug main discussion list Datum: 01.11.2017 15:43 Betreff:[Newsletter] Re: [gpfsug-discuss] Combine different rules - tip: use mmfind & co; FOR FILESET; FI

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Oesterlin, Robert
One thing that I’ve run into before is that on older file systems you had the “*.quota” files in the file system root. If you upgraded the file system to a newer version (so these files aren’t used) - There was a bug at one time where these didn’t get properly migrated during a restripe. Solutio

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
Thanks all for the suggestions. Having our metadata NSDs fill up was what prompted this exercise, but space was previously feed up on those by switching them from metadata+data to metadataOnly and using a policy to migrate files out of that pool. So these now have about 30% free space (more if you

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread David Johnson
One thing that may be relevant is if you have snapshots, depending on your release level, inodes in the snapshot may considered immutable, and will not be migrated. Once the snapshots have been deleted, the inodes are freed up and you won’t see the (somewhat misleading) message about no space.

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
We have no snapshots ( they were the first to go when we initially hit the full metadata NSDs). I've increased quotas so that no filesets have hit a space quota. Verified that there are no inode quotas anywhere. mmdf shows the least amount of free space on any nsd to be 9% free. Still getting t

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Scott Fadden
I opened a defect on this the other day, in my case it was an incorrect error message. What it meant to say was,"The pool is not empty." Are you trying to remove the last disk in a pool? If so did you empty the pool with a MIGRATE policy first?      Scott FaddenSpectrum Scale - Technical MarketingP

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Scott Fadden
Sorry just reread as I hit send and saw this was mmrestripe, in my case it was mmdeledisk.   Did you try running the command on just one pool. Or using -B instead?   What is the file it is complaining about in "/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779711" ?   Looks like it could be relate

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
Hmm, this sounds suspicious. We have 10 NSDs in a pool called system. These were previously set to data+metaData with a policy that placed our home directory filesets on this pool. A few weeks ago the NSDs in this pool all filled up. To remedy that I 1. removed old snapshots 2. deleted some old ho

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
Addendum to last message: We haven't upgraded recently as far as I know (I just inherited this a couple of months ago.) but am planning an outage soon to upgrade from 4.2.0-4 to 4.2.3-5. My growing collection of output files generally contain something like This inode list was generated in the P

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Scott Fadden
You should open a PMR on this one.    You can see what file this is using (You may have to build the tools in the directory)   /usr/lpp/mmfs/samples/util/tsfindinode  53506   This will at least tell you what kind of file it is. Though I am guessing this is a red herring.    Scott FaddenSpectrum Sca

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Frederick Stock
Assuming you are replicating data and metadata have you confirmed that all failure groups have the same free space? That is could it be that one of your failure groups has less space than the others? You can verify this with the output of mmdf and look at the NSD sizes and space available. Fr

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
We do have different amounts of space in the system pool which had the changes applied: [root@scg4-hn01 ~]# mmdf gsfs0 -P system diskdisk size failure holdsholds free KB free KB namein KBgroup metadata datain full blocks

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread Frederick Stock
Did you run the tsfindinode command to see where that file is located? Also, what does the mmdf show for your other pools notably the sas0 storage pool? Fred __ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 sto...@us.ibm.com From: John Hanks

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
tsfindiconde tracked the file to user.quota, which somehow escaped my previous attempt to "mv *.quota /elsewhere/" I've moved that now and verified it is actually gone and will retry once the current restripe on the sata0 pool is wrapped up. jbh On Thu, Nov 2, 2017 at 10:57 AM, Frederick Stock w

Re: [gpfsug-discuss] mmrestripefs "No space left on device"

2017-11-02 Thread John Hanks
Yep, looks like Robert Oesterlin was right, it was the old quota files causing the snag. Now sure how "mv *.quota" managed to move the group file and not the user file, but I'll let that remain a mystery of the universe. In any case I have a restripe running now and have learned a LOT about all the