On Sat, Feb 28, 2009 at 18:13:12 PST 2009, Joel Becker wrote:
> On Sat, Feb 28, 2009 at 12:09:37PM +0000, Nuno Fernandes wrote:
> > > That's rather odd. What is the blocksize and cluster size of
> > > this filesystem? Can you send me the output of 'debugfs.ocfs2 -R "stat
> > > /.zbr" /dev/hda3'?
> > >
> > > Joel
> >
> > Hi,
> >
> > Well.. we've deleted over 275 000 files and the problem disappeared... at
> > least for now. We'll just have to wait for that amount of files to be added.
>
> I wanted to see what .zbr looked like, because my hunch was that
> you were very fragmented and couldn't allocate enough contiguous space
> to create more extent blocks. Deleting files would perhaps have freed
> some in-use extent blocks, thus making it work again.
> If you run into the problem again, we'll revisit.
I have this happen on different file systems running different mainline kernel
versions created with different versions of ocfs2-tools. I've seen 1 file
system "fill up" with only 40% odd in use according to df. What's interesting
is that while 1 node can't write anymore, another node (only 2 nodes mounted
rw) will still be able to write to the FS, but neither will be able to
completely fill the FS.
I have created new ocfs2 file systems and the problem doesn't seem to occur (or
it gets very close be actually being full), but weeks/months later after being
in use, the problem develops. The FS I mentioned above that filled up at ~40%
use was about 6 months old and you could say it did it's fair share of "work".
I just dumped some 1GB files in a directory using dd to fill up a FS (full at
87%) to that I could collect some debugfs.ocfs2 information. Upon not being
able to create any further 1GB files (operations on the FS got really slow), I
successfully created some 10MB files. I then created 100+ 1MB files using dd.
What seemed strange is that dd kept returning "no space left on device", but it
would always write 995328 bytes of data to the file.
dd if=/dev/zero of=2156 bs=1k count=1k
dd: writing `2156': No space left on device
973+0 records in
972+0 records out
995328 bytes (995 kB) copied, 0.0551648 s, 18.0 MB/s
dd if=/dev/zero of=2157 bs=1k count=1k
dd: writing `2157': No space left on device
973+0 records in
972+0 records out
995328 bytes (995 kB) copied, 0.0545104 s, 18.3 MB/s
echo "stats" | debugfs.ocfs2 /dev/xvdd1 |cat
debugfs.ocfs2 1.4.3
debugfs: Revision: 0.90
Mount Count: 0 Max Mount Count: 20
State: 0 Errors: 0
Check Interval: 0 Last Check: Thu Oct 8 20:23:54 2009
Creator OS: 0
Feature Compat: 0
Feature Incompat: 0
Tunefs Incomplete: 0
Feature RO compat: 0
Root Blknum: 5 System Dir Blknum: 6
First Cluster Group Blknum: 3
Block Size Bits: 12 Cluster Size Bits: 12
Max Node Slots: 16
Extended Attributes Inline Size: 0
Label:
UUID: BD88053E263B4CEAABC941FE3413B4EA
UUID_hash: 0 (0x0)
Cluster stack: classic o2cb
Inode: 2 Mode: 00 Generation: 964566170 (0x397e1c9a)
FS Generation: 964566170 (0x397e1c9a)
CRC32: 00000000 ECC: 0000
Type: Unknown Attr: 0x0 Flags: Valid System Superblock
Dynamic Features: (0x0)
User: 0 (root) Group: 0 (root) Size: 0
Links: 0 Clusters: 13107199
ctime: 0x48f80fe8 -- Fri Oct 17 15:09:12 2008
atime: 0x0 -- Thu Jan 1 10:00:00 1970
mtime: 0x48f80fe8 -- Fri Oct 17 15:09:12 2008
dtime: 0x0 -- Thu Jan 1 10:00:00 1970
ctime_nsec: 0x00000000 -- 0
atime_nsec: 0x00000000 -- 0
mtime_nsec: 0x00000000 -- 0
Last Extblk: 0
Sub Alloc Slot: Global Sub Alloc Bit: 65535
debugfs:
echo "stat /brad/aaaaa" | debugfs.ocfs2 /dev/xvdd1 |cat
debugfs.ocfs2 1.4.3
debugfs: Inode: 11539619 Mode: 0750 Generation: 3859452415
(0xe60a91ff)
FS Generation: 964566170 (0x397e1c9a)
CRC32: 00000000 ECC: 0000
Type: Directory Attr: 0x0 Flags: Valid
Dynamic Features: (0x0)
User: 1000 (brad) Group: 2000 (unknown) Size: 4096
Links: 2 Clusters: 1
ctime: 0x4b28b899 -- Wed Dec 16 21:38:17 2009
atime: 0x4b28b03b -- Wed Dec 16 21:02:35 2009
mtime: 0x4b28b899 -- Wed Dec 16 21:38:17 2009
dtime: 0x0 -- Thu Jan 1 10:00:00 1970
ctime_nsec: 0x29bd9e6e -- 700292718
atime_nsec: 0x31a68876 -- 832997494
mtime_nsec: 0x29bd9e6e -- 700292718
Last Extblk: 0
Sub Alloc Slot: 2 Sub Alloc Bit: 87
Tree Depth: 0 Count: 243 Next Free Rec: 1
## Offset Clusters Block# Flags
0 0 1 9343355 0x0
debugfs:
Hopefully this sheds some light on what's going on. Please let me know if you
need any further info.
Cheers,
Brad
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
