Re: files of size larger than fs size
>The problem appears to be mixing calls to lseek64 with calls to fread >and fwrite. Oh, of course. I didn't see that. You can't use the file descriptor of a file that is opened as a stream. This test case uses the fileno() function to mess with the internals of the stream. fseeko64() is the proper function to position a stream. - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
On Thu, 2005-03-17 at 13:48 -0800, Max wrote: > Dave, > > Shouldn't "fread(&data,sizeof(data),1,f)" and "read(fn, &data, > sizeof(data))" produce identical results? > Is it a bug or what? fseek(f, offset, SEEK_SET); fread(&data, sizeof(data, 1, f); should produce identical results to lseek(fn, offset, SEEK_SET); read(fn, &data, sizeof(data); However lseek(fn, offset, SEEK_SET); fread(&data, sizeof(data, 1, f); isn't valid. -- David Kleikamp IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
Dave,
Shouldn't "fread(&data,sizeof(data),1,f)" and "read(fn, &data,
sizeof(data))" produce identical results?
Is it a bug or what?
Max
Dave Kleikamp wrote:
With this change, the file size on jfs becomes 2^48 + 4 as expected.
--- jfs_bug.c.orig 2005-03-17 14:18:48.229634648 -0600
+++ jfs_bug.c 2005-03-17 15:32:45.952750104 -0600
@@ -13,12 +13,14 @@ int data = 0;
struct flock fl;
void read1() {
-size_t rc = fread(&data,sizeof(data),1,f);
+/* size_t rc = fread(&data,sizeof(data),1,f); */
+size_t rc = read(fn, &data, sizeof(data));
printf("read() rc = %llu\n",rc);
}
void write1() {
-size_t rc = fwrite(&data,sizeof(data),1,f);
+/* size_t rc = fwrite(&data,sizeof(data),1,f); */
+size_t rc = write(fn, &data, sizeof(data));
printf("write() rc = %llu\n",rc);
}
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
With this change, the file size on jfs becomes 2^48 + 4 as expected.
--- jfs_bug.c.orig 2005-03-17 14:18:48.229634648 -0600
+++ jfs_bug.c 2005-03-17 15:32:45.952750104 -0600
@@ -13,12 +13,14 @@ int data = 0;
struct flock fl;
void read1() {
-size_t rc = fread(&data,sizeof(data),1,f);
+/* size_t rc = fread(&data,sizeof(data),1,f); */
+size_t rc = read(fn, &data, sizeof(data));
printf("read() rc = %llu\n",rc);
}
void write1() {
-size_t rc = fwrite(&data,sizeof(data),1,f);
+/* size_t rc = fwrite(&data,sizeof(data),1,f); */
+size_t rc = write(fn, &data, sizeof(data));
printf("write() rc = %llu\n",rc);
}
--
David Kleikamp
IBM Linux Technology Center
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
On Thu, 2005-03-17 at 14:51 -0600, Dave Kleikamp wrote: > On Thu, 2005-03-17 at 12:06 -0800, Bryan Henderson wrote: > > >I found > > >that for larger values, your test program is returning -1, but unsigned > > >it appears as 18446744073709551615. > > > > You mean you ran it? Then what about the more interesting question of > > what your filesize ends up to be? You say JFS allows files up to 2**52 > > bytes, so I expect the test case would succeed up through the write at > > 2**48 and leave the filesize 2**48 + 8. But Max reports seeing 2**48 - > > 4080. > > Yeah, this is funny. I missed that part of the story. I'll try to > figure out what is going on here. The problem appears to be mixing calls to lseek64 with calls to fread and fwrite. fread & fwrite don't appear to honor the position set by lseek64. I assume replacing fread & fwrite with read & write will fix it. I haven't tried it yet, but I'm about to. -- David Kleikamp IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
On Thu, 2005-03-17 at 12:06 -0800, Bryan Henderson wrote: > >I found > >that for larger values, your test program is returning -1, but unsigned > >it appears as 18446744073709551615. > > You mean you ran it? Then what about the more interesting question of > what your filesize ends up to be? You say JFS allows files up to 2**52 > bytes, so I expect the test case would succeed up through the write at > 2**48 and leave the filesize 2**48 + 8. But Max reports seeing 2**48 - > 4080. Yeah, this is funny. I missed that part of the story. I'll try to figure out what is going on here. > It's conceivable that the reporting of the filesize is wrong, by the way. jfs_debugfs tells me the same thing. So ls has it right. Thanks, Shaggy -- David Kleikamp IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
>I found >that for larger values, your test program is returning -1, but unsigned >it appears as 18446744073709551615. You mean you ran it? Then what about the more interesting question of what your filesize ends up to be? You say JFS allows files up to 2**52 bytes, so I expect the test case would succeed up through the write at 2**48 and leave the filesize 2**48 + 8. But Max reports seeing 2**48 - 4080. It's conceivable that the reporting of the filesize is wrong, by the way. - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
On Wed, 2005-03-16 at 20:52 -0800, Max wrote: > Bryan, > > I'm not experienced in filesystems. But I've derived the testcase > program from some buggy application that occasionally created huge > files on my fs. That was not so easy to reproduce since not every > sequence of seeks/reads/writes results in a huge file. But finally I > got it 100% reproducible. > > So I would appriciate if somebody make any good from that testcase > code. man lseek tells you to test the return code against (off_t)-1. I found that for larger values, your test program is returning -1, but unsigned it appears as 18446744073709551615. > > Different filesizes look strange to me. What I have so far: > > JFS:281474976706576 = 2**48 - 4080 > XFS: 72057594037923856 = 2**56 - 4080 > EXT3: 1099511627784 = 2**40 + 8 I can speak for jfs, and it supports file sizes up to 2**52. It uses 40 bits to address the logical block offset within a file with a 4K (2**12) size block. The individual file systems set sb->s_maxbytes to the appropriate value. > Thanks, > Max > > P.S. direct link to the testcase program: > http://bugzilla.kernel.org/attachment.cgi?id=4729 -- David Kleikamp IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
Bryan, I'm not experienced in filesystems. But I've derived the testcase program from some buggy application that occasionally created huge files on my fs. That was not so easy to reproduce since not every sequence of seeks/reads/writes results in a huge file. But finally I got it 100% reproducible. So I would appriciate if somebody make any good from that testcase code. Different filesizes look strange to me. What I have so far: JFS:281474976706576 = 2**48 - 4080 XFS: 72057594037923856 = 2**56 - 4080 EXT3: 1099511627784 = 2**40 + 8 Thanks, Max P.S. direct link to the testcase program: http://bugzilla.kernel.org/attachment.cgi?id=4729 Bryan Henderson wrote: But anyway it's interesting why the resulting sparse files have different size on different fs? That looks like a bug. Assuming you didn't see any seeks or writes fail, the file size on all filesystems should be 2^56 + 4. I suspect this is beyond the maximum file size allowed by the filesystem in some cases, so the write isn't happening, which means you should get a failure return code. In the results you showed, the filesize ends up being a little less than 2^48, which is not a place that you wrote ever. -- Bryan Henderson IBM Almaden Research Center San Jose CA Filesystems - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
>But anyway it's interesting why the resulting sparse >files have different size on different fs? That looks like a bug. Assuming you didn't see any seeks or writes fail, the file size on all filesystems should be 2^56 + 4. I suspect this is beyond the maximum file size allowed by the filesystem in some cases, so the write isn't happening, which means you should get a failure return code. In the results you showed, the filesize ends up being a little less than 2^48, which is not a place that you wrote ever. -- Bryan Henderson IBM Almaden Research Center San Jose CA Filesystems - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
Thanks for the explanation. I closed my bugreport as INVALID. But anyway it's ineteresting why the resulting sparse files have different size on different fs? Max Phillip Lougher wrote: Max wrote: Hello! I've discovered that it is possible to create files of size much larger than partition size. I thought that this is JFS bug, so I've filed a bugreport against it at http://bugzilla.kernel.org/show_bug.cgi?id=4345 Detailed info and testcase program are provided there. Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel. Could kernel gurus please investigate the problem? Your test case isn't writing a full file, it is only writing 4 bytes at various offsets (1^32, 1^40, 1^48, 1^56). The filesystems you mention support files with "holes" in them, in otherwords they support gaps between data which don't take up any storage. Even though your test case is creating a huge file, only a couple of bytes are written, the rest of the huge file doesn't take up any space. The behaviour you're seeing isn't a bug... Phillip Lougher - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
On Tue, Mar 15, 2005 at 06:17:52PM -0800, Max wrote: > Hello! > > I've discovered that it is possible to create files of size much larger > than partition size. > I thought that this is JFS bug, so I've filed a bugreport against it at > http://bugzilla.kernel.org/show_bug.cgi?id=4345 > Detailed info and testcase program are provided there. > > Later I've found that at least XFS and EXT3 filesystems have the same > problem (though the resulting filesize is different for each fs). So the > problem may be not in fs code but in some other piece of kernel. > > Could kernel gurus please investigate the problem? > > P.S. I'm using debian-amd64 with kernel 2.6.11.3. So I didn't check if > 32-bit kernel has this bug. This is not a bug but a fairly standard UNIX feature where you are creating sparse files. If you stat these files, you'll see the actual number of filesystem blocks allocated is much smaller than the apparent length of the file. For more info: google.com This one looks like a decent overview: http://www.lrdev.com/lr/unix/sparsefile.html Sonny - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: files of size larger than fs size
Max wrote: Hello! I've discovered that it is possible to create files of size much larger than partition size. I thought that this is JFS bug, so I've filed a bugreport against it at http://bugzilla.kernel.org/show_bug.cgi?id=4345 Detailed info and testcase program are provided there. Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel. Could kernel gurus please investigate the problem? Your test case isn't writing a full file, it is only writing 4 bytes at various offsets (1^32, 1^40, 1^48, 1^56). The filesystems you mention support files with "holes" in them, in otherwords they support gaps between data which don't take up any storage. Even though your test case is creating a huge file, only a couple of bytes are written, the rest of the huge file doesn't take up any space. The behaviour you're seeing isn't a bug... Phillip Lougher - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
files of size larger than fs size
Hello! I've discovered that it is possible to create files of size much larger than partition size. I thought that this is JFS bug, so I've filed a bugreport against it at http://bugzilla.kernel.org/show_bug.cgi?id=4345 Detailed info and testcase program are provided there. Later I've found that at least XFS and EXT3 filesystems have the same problem (though the resulting filesize is different for each fs). So the problem may be not in fs code but in some other piece of kernel. Could kernel gurus please investigate the problem? P.S. I'm using debian-amd64 with kernel 2.6.11.3. So I didn't check if 32-bit kernel has this bug. Max - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
