Re: [Veritas-vx] vxfs filesystem full - can not expand.
Hi all, I agree, there should be no special cases for what should be a routine filesystem expansion. UFS can be grown any time, not matter what the filesystem internals happen to be at the time of 100% fs full bonanza. Even with sufficient monitoring in place, it's a bit hard to pickup and prevent fastly growing filesystem usage from hitting the 100% limit and stopping an application component. And it's especially interesting seeing lots of SAN storage free on the DG only to realise none of it can be used for growing the filesystem online. Scott, many thanks for the investigation into this. Seems we're not the only ones facing this situation. We've grown the fs via other means (application process freed up a bit of space eventually from one of the files it was writing to, took a few hours). Here's what we were able to do after growing the actual volume & above process freeing the space up: bash-2.05# fstyp -v /dev/vx/dsk/san/vol vxfs magic a501fcf5 version 7 ctime Fri May 18 11:19:20 2007 logstart 0 logend 0 bsize 8192 size 5288220 dsize 0 ninode 5288220 nau 0 defiextsize 0 ilbsize 0 immedlen 96 ndaddr 10 aufirst 0 emap 0 imap 0 iextop 0 istart 0 bstart 0 femap 0 fimap 0 fiextop 0 fistart 0 fbstart 0 nindir 2048 aulen 32768 auimlen 0 auemlen 1 auilen 0 aupad 0 aublocks 32768 maxtier 15 inopb 32 inopau 0 ndiripau 0 iaddrlen 1 bshift 13 inoshift 5 bmask e000 boffmask 1fff checksum eb5643da oltext1 11 oltext2 8708 oltsize 1 checksum2 0 free 376102 ifree 0 efree 8182 8580 8056 6205 4586 2886 1612 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 bash-2.05# df -k . Filesystemkbytesused avail capacity Mounted on /dev/vx/dsk/san/vol 42305760 39291760 299355293%/san/vol bash-2.05# /usr/lib/fs/vxfs/fsadm -F vxfs -b 86708672 /san/vol UX:vxfs fsadm: INFO: V-3-25942: /dev/vx/rdsk/san/vol size increased from 84611520 sectors to 86708672 sectors bash-2.05# df -k /san/vol Filesystemkbytesused avail capacity Mounted on /dev/vx/dsk/san/vol 43354336 39291792 403390491%/san/vol We've got a case open with Symantec, I'll email it separately... if I can still find it that is... -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
On Tue, 18 Mar 2008, Rajiv Gunja wrote: > ""Enhancements were a couple versions ago to reserve space internally within > the file system so users wouldn't have to manually reserve free space. > See:"" > > Scott, I read through that article, but I do not think that is truly the > case. We have over 4000 Solaris servers in production running Veritas > version 4.0 to 4.1 with various MP versions (mostly mp3). I have seen at > least 2 or 3 cases in the last year alone when a FS was at 100%, I was > unable to resize the FS. > > Yes I know the article says "" In very rare cases, a 100% full file system > may not be grown."" > > But it is not so rare. Of course it is very rare for us to see 100% on a FS > in production as we have monitors in place, but still when there is > unforeseen increase, vxresize has failed. Funny, I was just in SFRAC class, and ran into this same issue and asked about reserved space. It would see that in MANY cases you cannot expand it once at 100%. This is a bug IMHO. Need to be able to either reserve space to prevent 100% full, or resize at 100%. Jason -- Jason Slagle /"\ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . \ / ASCII Ribbon Campaign . X - NO HTML/RTF in e-mail . / \ - NO Word docs in e-mail . ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
""Enhancements were a couple versions ago to reserve space internally within the file system so users wouldn't have to manually reserve free space. See:"" Scott, I read through that article, but I do not think that is truly the case. We have over 4000 Solaris servers in production running Veritas version 4.0 to 4.1 with various MP versions (mostly mp3). I have seen at least 2 or 3 cases in the last year alone when a FS was at 100%, I was unable to resize the FS. Yes I know the article says "" In very rare cases, a 100% full file system may not be grown."" But it is not so rare. Of course it is very rare for us to see 100% on a FS in production as we have monitors in place, but still when there is unforeseen increase, vxresize has failed. -GGR Rajiv G Gunja ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
OK, got some more info. If the file system is a multiple of the AU size, then VxFS can hit this condition. So, in terms of future avoidance >>always create file systems with 1 block more than a multiple of 2^15 blocks >>(and apply the same restriction when growing or shrinking file systems), then >>we'll always have enough hidden space to grow the fs. Note: that's file system block size which is 1k (default), 2k, 4k, or 8k. My source also offered some possibilities on why even removing files didn't allow them to grow. 1. this may be because we need to allocate 8k extents as part of growing an fs and the files he removed didn't have any 8k or larger extents. (if that's the case then he would be able to make space to grow by removing files with larger extents*.) I'm sorry I didn't think of his second suggestion immediately, as it happens not infrequently: 2. or maybe he had a checkpoint and the blocks for the files he removed just got transferred to the ckpt, in which case he would need to remove the ckpt *You can get some extent information for an individual file from fsadm_vxfs, e.g.,: $ <mailto:[EMAIL PROTECTED]$> fsadm -t vxfs -E -f mynovel.txt Extent Fragmentation Report TotalAverage Average Total FilesFile Blks# Extents Free Blks 1 84 4 211974585 ... The free blocks information is for the file system as a whole. With 84 blocks and 4 extents, one of them must be at least 8k in this example. (the above example is -t for linux; use -F for Solaris|HPUX or -V for AIX) Regards, Scott From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Scott Kaiser Sent: Monday, March 17, 2008 1:29 PM To: Rajiv Gunja; Sengor Cc: Asim Zuberi; veritas-vx@mailman.eng.auburn.edu; Doug Hughes Subject: Re: [Veritas-vx] vxfs filesystem full - can not expand. Inodes are dynamically allocated in VxFS, so that's not likely to be the cause (unless you have a multiple billion inodes or a very, very old disk layout). Enhancements were a couple versions ago to reserve space internally within the file system so users wouldn't have to manually reserve free space. See: http://seer.entsupport.symantec.com/docs/249933.htm But you said you're running on version 5. So it sounds like you've hit one of the 'rare cases' What is the exact size of the file system? I seem to recall the rare cases involved being a multiple of some number. Regards, Scott From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rajiv Gunja Sent: Monday, March 17, 2008 11:10 AM To: Sengor Cc: Asim Zuberi; Doug Hughes; veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxfs filesystem full - can not expand. -- That is very odd. How are you doing on i-nodes? Too many / any free left? -- RM fails, how about mv (move) there should be a way to free up i-nodes and or space. -GGR On Sun, Mar 16, 2008 at 11:31 PM, Sengor <[EMAIL PROTECTED]> wrote: On 3/17/08, Asim Zuberi <[EMAIL PROTECTED]> wrote: > Did you consider doing "fsck -F vxfs" on the filesystem? Nope, since we need the fs to remain online. -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
Inodes are dynamically allocated in VxFS, so that's not likely to be the cause (unless you have a multiple billion inodes or a very, very old disk layout). Enhancements were a couple versions ago to reserve space internally within the file system so users wouldn't have to manually reserve free space. See: http://seer.entsupport.symantec.com/docs/249933.htm But you said you're running on version 5. So it sounds like you've hit one of the 'rare cases' What is the exact size of the file system? I seem to recall the rare cases involved being a multiple of some number. Regards, Scott From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rajiv Gunja Sent: Monday, March 17, 2008 11:10 AM To: Sengor Cc: Asim Zuberi; Doug Hughes; veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxfs filesystem full - can not expand. -- That is very odd. How are you doing on i-nodes? Too many / any free left? -- RM fails, how about mv (move) there should be a way to free up i-nodes and or space. -GGR On Sun, Mar 16, 2008 at 11:31 PM, Sengor <[EMAIL PROTECTED]> wrote: On 3/17/08, Asim Zuberi <[EMAIL PROTECTED]> wrote: > Did you consider doing "fsck -F vxfs" on the filesystem? Nope, since we need the fs to remain online. -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
-- That is very odd. How are you doing on i-nodes? Too many / any free left? -- RM fails, how about mv (move) there should be a way to free up i-nodes and or space. -GGR On Sun, Mar 16, 2008 at 11:31 PM, Sengor <[EMAIL PROTECTED]> wrote: > On 3/17/08, Asim Zuberi <[EMAIL PROTECTED]> wrote: > > Did you consider doing "fsck -F vxfs" on the filesystem? > > Nope, since we need the fs to remain online. > > -- > _/ sengork.blogspot.com / > ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
On 3/17/08, Asim Zuberi <[EMAIL PROTECTED]> wrote: > Did you consider doing "fsck -F vxfs" on the filesystem? Nope, since we need the fs to remain online. -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
Did you consider doing "fsck -F vxfs" on the filesystem?
=]-Original Message-
=]From: [EMAIL PROTECTED]
=][mailto:[EMAIL PROTECTED]
=]On Behalf Of Sengor
=]Sent: Sunday, March 16, 2008 7:36 PM
=]To: Rajiv Gunja
=]Cc: veritas-vx@mailman.eng.auburn.edu; Doug Hughes
=]Subject: Re: [Veritas-vx] vxfs filesystem full -
=] can not expand.
=]
=]On 3/17/08, Rajiv Gunja <[EMAIL PROTECTED]> wrote:
=]> Here is a trick I do. Find out the sizes of
=]directories which are
=]> overly using space. Pick one which will bring the
=]usage DOWN by 10%.
=]> Now tar/cpio the whole directory to a different
=]partition and delete
=]> it.(of course logically, that directory should
=]not be used by any of
=]> the applications actively. If they are, shutdown
=]the app). Now expand
=]> it to whatever size you want to the hearts
=]content. Veritas needs 5 to
=]> 10% of free space to do the actual expansion.
=]
=]Tried that already, the issue with that approach
=]was vxfs would not even let any files be deleted or
=]zeroed out from the affected filesystem... So rm
=]-fr and cat /dev/null would all fail with "no space
=]left" reasons.
=]
=]
=]--
=]_/
=]sengork.blogspot.com /
=]
=]___
=]Veritas-vx maillist -
=]Veritas-vx@mailman.eng.auburn.edu
=]http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
=]
___
Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
On 3/17/08, Rajiv Gunja <[EMAIL PROTECTED]> wrote: > Here is a trick I do. Find out the sizes of directories which are overly > using space. Pick one which will bring the usage DOWN by 10%. Now tar/cpio > the whole directory to a different partition and delete it.(of course > logically, that directory should not be used by any of the applications > actively. If they are, shutdown the app). Now expand it to whatever size you > want to the hearts content. Veritas needs 5 to 10% of free space to do the > actual expansion. Tried that already, the issue with that approach was vxfs would not even let any files be deleted or zeroed out from the affected filesystem... So rm -fr and cat /dev/null would all fail with "no space left" reasons. -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
Here is a trick I do. Find out the sizes of directories which are overly using space. Pick one which will bring the usage DOWN by 10%. Now tar/cpio the whole directory to a different partition and delete it.(of course logically, that directory should not be used by any of the applications actively. If they are, shutdown the app). Now expand it to whatever size you want to the hearts content. Veritas needs 5 to 10% of free space to do the actual expansion. Good luck and let us how you fare. -GGR Rajiv G Gunja On Sat, Mar 15, 2008 at 10:11 PM, Sengor <[EMAIL PROTECTED]> wrote: > On 3/15/08, Doug Hughes <[EMAIL PROTECTED]> wrote: > > You can still expand it, just pick a small number.. a block or two.. > > that gives you some space to expand it slightly more, and then slightly > > more, and so in. It's not ideal, but it's a known issue and it works. > > grow by a few bytes and then double and double and double until you get > > where you want it to be. > > Tried doing that as well, but not on such small quantities, lowest we > went to was about few MB. In either case I found this quite > surprising/dissapointing about vxfs. > > Thanks for the tip, I'll keep it in mind for a certain next time > encounter. > > -- > _/ sengork.blogspot.com / > > ___ > Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx > ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
On 3/15/08, Doug Hughes <[EMAIL PROTECTED]> wrote: > You can still expand it, just pick a small number.. a block or two.. > that gives you some space to expand it slightly more, and then slightly > more, and so in. It's not ideal, but it's a known issue and it works. > grow by a few bytes and then double and double and double until you get > where you want it to be. Tried doing that as well, but not on such small quantities, lowest we went to was about few MB. In either case I found this quite surprising/dissapointing about vxfs. Thanks for the tip, I'll keep it in mind for a certain next time encounter. -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxfs filesystem full - can not expand.
Sengor wrote: > Hi colleagues, > > Recently we've had a situation where a mounted vxfs filesystem was @ > 100% usage (no free space) on Solaris 9, SF5. So we thought "great > we'll go and increase the volume & the filesystem and be done with > it". > > To our unpleasant surprise fsadm returned error saying filesystem can > not be expanded due to it being full (even though volume has been > expanded already). We tried deleting some files from the affected fs, > to no avail. As root, even rm & trying to zero out some of the files > simply refused to work due to the filesystem being full. > > Now from what I understand there's no such thing as minfree root > reserved space for vxfs, which would allow the fs to be expanded once > it's full. > > I'm quite disappointed to see vxfs not reserve any space on the side > for it's own administrative purposes, since fs being @ 100% is when > you need to expand the filesystem the most, out of any other case > scenarios. > > How do people go about resolving or avoiding the issue of not being > able to expand the filesystem once it's full? > > Are there any precautions/safeguards to consider when the fs is being > created initially? Perhaps some means of reserving administrative fs > space (without quotas) ? > > Thank you. > > You can still expand it, just pick a small number.. a block or two.. that gives you some space to expand it slightly more, and then slightly more, and so in. It's not ideal, but it's a known issue and it works. grow by a few bytes and then double and double and double until you get where you want it to be. ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
[Veritas-vx] vxfs filesystem full - can not expand.
Hi colleagues, Recently we've had a situation where a mounted vxfs filesystem was @ 100% usage (no free space) on Solaris 9, SF5. So we thought "great we'll go and increase the volume & the filesystem and be done with it". To our unpleasant surprise fsadm returned error saying filesystem can not be expanded due to it being full (even though volume has been expanded already). We tried deleting some files from the affected fs, to no avail. As root, even rm & trying to zero out some of the files simply refused to work due to the filesystem being full. Now from what I understand there's no such thing as minfree root reserved space for vxfs, which would allow the fs to be expanded once it's full. I'm quite disappointed to see vxfs not reserve any space on the side for it's own administrative purposes, since fs being @ 100% is when you need to expand the filesystem the most, out of any other case scenarios. How do people go about resolving or avoiding the issue of not being able to expand the filesystem once it's full? Are there any precautions/safeguards to consider when the fs is being created initially? Perhaps some means of reserving administrative fs space (without quotas) ? Thank you. -- _/ sengork.blogspot.com / ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
