Re: very big files on cd9660 file system
Bruce Evans [EMAIL PROTECTED] writes: Mostly (b). Sizes are 64 bits in the standard, but FreeBSD has always silently discarded the highest 32 bits and corrupted the next highest bit to a sign bit, so the file size limit is at most 2GB or 4GB (depending on whether the sign bit gets corrupted back to a value bit). ISO9660 does not use 64-bit values. Those 8-byte values you see in the headers are 32-bit values stored first in little-endian format and second in big-endian format. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: very big files on cd9660 file system
The attached patch should make the isonum functions in iso.h much
clearer. It also gets rid of the optimizated versions; I trust the
compiler to take care of that.
The inode number situation can be improved by dividing the byte offset
of the directory entry by a suitable number guaranteed not to be
larger than the size in bytes of any directory entry, e.g. 32 (IIUC, a
directory entry contains at least 33 bytes of metadata and padding in
addition to the file name)
DES
--
Dag-Erling Smørgrav - [EMAIL PROTECTED]
Index: sys/isofs/cd9660/iso.h
===
RCS file: /home/ncvs/src/sys/isofs/cd9660/iso.h,v
retrieving revision 1.30
diff -u -r1.30 iso.h
--- sys/isofs/cd9660/iso.h 14 Mar 2005 13:22:41 - 1.30
+++ sys/isofs/cd9660/iso.h 25 Aug 2005 12:33:44 -
@@ -277,79 +277,65 @@
* outside the kernel. Thus we don't hide them here.
*/
-static __inline int isonum_711(u_char *);
-static __inline int
-isonum_711(p)
- u_char *p;
+/*
+ * 7xy
+ * x - 1 = 8 bits, 2 = 16 bits, 3 = 32 bits
+ * y - 1 = little-endian, 2 = big-endian, 3 = both (le then be)
+ */
+
+static __inline uint8_t
+isonum_711(unsigned char *p)
{
- return *p;
+ return p[0];
}
-static __inline int isonum_712(char *);
-static __inline int
-isonum_712(p)
- char *p;
+static __inline uint8_t
+isonum_712(unsigned char *p)
{
- return *p;
+ return p[0];
}
-#ifndef UNALIGNED_ACCESS
-
-static __inline int isonum_723(u_char *);
-static __inline int
-isonum_723(p)
- u_char *p;
+static __inline uint8_t
+isonum_713(unsigned char *p)
{
- return *p|(p[1] 8);
+ return p[0];
}
-static __inline int isonum_733(u_char *);
-static __inline int
-isonum_733(p)
- u_char *p;
+static __inline uint16_t
+isonum_721(unsigned char *p)
{
- return *p|(p[1] 8)|(p[2] 16)|(p[3] 24);
+ return (p[0] | p[1] 8);
}
-#else /* UNALIGNED_ACCESS */
-
-#if BYTE_ORDER == LITTLE_ENDIAN
-
-static __inline int
-isonum_723(p)
- u_char *p
+static __inline uint16_t
+isonum_722(unsigned char *p)
{
- return *(u_int16t *)p;
+ return (p[1] | p[0] 8);
}
-static __inline int
-isonum_733(p)
- u_char *p;
+static __inline uint16_t
+isonum_723(unsigned char *p)
{
- return *(u_int32t *)p;
+ return (p[0] | p[1] 8);
}
-#endif
-
-#if BYTE_ORDER == BIG_ENDIAN
-
-static __inline int
-isonum_723(p)
- u_char *p
+static __inline uint32_t
+isonum_731(unsigned char *p)
{
- return *(u_int16t *)(p + 2);
+ return (p[0] | p[1] 8 | p[2] 16 | p[3] 24);
}
-static __inline int
-isonum_733(p)
- u_char *p;
+static __inline uint32_t
+isonum_732(unsigned char *p)
{
- return *(u_int32t *)(p + 4);
+ return (p[3] | p[2] 8 | p[1] 16 | p[0] 24);
}
-#endif
-
-#endif /* UNALIGNED_ACCESS */
+static __inline uint32_t
+isonum_733(unsigned char *p)
+{
+ return (p[0] | p[1] 8 | p[2] 16 | p[3] 24);
+}
/*
* Associated files have a leading '='.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: very big files on cd9660 file system
ISO9660 does not use 64-bit values. Those 8-byte values you see in the headers are 32-bit values stored first in little-endian format and second in big-endian format. So, in my original question, the blame lies solely with 3) ISO-9660 standard ? No single file on a ISO9660 filesystem may exceed 4Gb? Is there some newer, superceeding backwards-compatible standard -- all the new DVD devices are now offering the media to store large files? Or is fat32 the only cross-platform option today? -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: very big files on cd9660 file system
Mikhail Teterin [EMAIL PROTECTED] writes: No single file on a ISO9660 filesystem may exceed 4Gb? The ISO 9660 file system was designed for a storage medium which had a fixed capacity of 600 MB. Is there some newer, superceeding backwards-compatible standard -- all the new DVD devices are now offering the media to store large files? Or is fat32 the only cross-platform option today? You're supposed to use UDF on DVDs. I don't think it's backward compatible with ISO 9660. I suspect that the reason why ISO 9660 is being used on DVDs is compatibility with BIOSes which support booting from ISO 9660 but not (yet) UDF. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: very big files on cd9660 file system
On Fri, 19 Aug 2005, Mikhail Teterin wrote:
I have a cd9660 image with several files on it. One of the files is very large
(above 4Gb). When I mount the image, the size of this file is shown as
realsize % 4Gb -- 758876749 bytes instead of 5053844045.
What should I blame:
1) The software, that created the image (modified mkisofs)
2) cd9660 part of the FreeBSD kernel
3) ISO-9660 standard
Mostly (b). Sizes are 64 bits in the standard, but FreeBSD has always
silently discarded the highest 32 bits and corrupted the next highest
bit to a sign bit, so the file size limit is at most 2GB or 4GB
(depending on whether the sign bit gets corrupted back to a value bit).
From cd9660_vfsops.c:
% ip-i_size = isonum_733(isodir-size);
This reads the size from the directory entry.
From iso.h:
% u_char size [ISODCL (11, 18)]; /* 733 */
This says that the size is in bytes 11-18 (option base 1) in the directory
entry. All 733 entries are 8 bytes. The others are for other sizes and
the extent (the starting block number for a file).
% static __inline int
% isonum_733(p)
% u_char *p;
% {
% return *p|(p[1] 8)|(p[2] 16)|(p[3] 24);
% }
This says that the the highest 32 bits are discarded for all 733 entries
and the sign bit in p[3] is corrupted, first by shifting it and then by
assigning the result to an int.
i_size has type long, unlike in most file systems in FreeBSD where it is
uint64_t or uint32_t, so I think the sign bit stays corrupted but doesn't
cause further problems by being converted to 33 top unsigned bits, giving
a limit of 2GB.
The file size limit is hit before the others. 31-bit block numbers with
2K-blocks work up to 4TB. There are likely to be overflow bugs at 1TB
before the 4TB limit is hit.
We still have the even closer limit of 4GB on media sizes. From
cd9660_node.c:
% ino_t
% isodirino(isodir, imp)
% struct iso_directory_record *isodir;
% struct iso_mnt *imp;
% {
% ino_t ino;
%
% ino = (isonum_733(isodir-extent) + isonum_711(isodir-ext_attr_length))
% imp-im_bshift;
% return (ino);
% }
This fakes the inode number as the byte offset of the directory entry.
ino_t is uint32_t, so this fails if the byte offset exceeds 4GB. The
eventual 32nd bit overflows to become a sign bit in the shift but then
gets overflows back to a correct bit in the assignent, so offsets
between 2GB and 4GB work accidentally.
Since the limit is on the offsets of directory entries, media larger
than 4GB can be used for cd9660 under FreeBSD iff all directroy entries
are below the limit, which happens automatically for the non-multi-session
case only.
See revs.1.77 and 1.99 for other bugs caused by isodirino().
Bruce
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
