Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc
Hello Theodore, Theodore Tso schrieb am Fri 29. Sep, 17:15 (-0400): If the filesystem is empty (or at least no no hashtree directories), then when the kernel creates new directories and expands to the point where they become indexed, they will be indexed with the PPC variant of the hash algorithm. This will be self consistent, and everything will work fine --- until the filesystem gets corrupted to the point where e2fsprogs needs to rebuild one or more hashed directories. At that point the directories will be rebuilt using the same conventions used by all other conventions, but the directories will no longer be useful on the PPC kernel. Joerg, can you confirm this? Yes. On a PPC machine, can you create a smallish ext3 filesystem (say, 4-8 megabytes), create a directory with enough files in it that it becomes indexed (verify using lsattr), and show that it works just fine on a PPC. Now take that image, and transfer it to an x86 machine; I don't have a x86 machine. you should find that the kernel can't look up any of the directories on the x86 machine. If you then run e2fsck -fD on that filesystem (running the e2fsck on either x86 or PPC; it shouldn't make a difference), then the resulting filesystem should work just fine on the x86, and fail on the PPC. I put two files at http://www.minet.uni-jena.de/~joergs/ img-broken and img-working. In the last image, img-working, I can access all files, especially test/broken fürß. The first image, img-broken, is the image after running e2fsck -Df on img-working. So you can test if one of these images work on x86, especially if the file test/broken fürß is accessable. Bye, Jörg. -- Wer A sagt, muß nicht B sagen. Er kann auch erkennen, daß A falsch war. (Erich Kästner) pgpIyLAtwyqgA.pgp Description: PGP signature
Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc
On Sep 28, 2006 21:08 -0400, Theodore Tso wrote: On Wed, Sep 27, 2006 at 02:22:51PM +0200, Jörg Sommer wrote: Package: e2fsprogs Version: 1.39-1 Severity: important you set the compiler option -fsigned-char, but on PowerPC the default is unsigned char. This makes the kernel uses unsigned and e2fsck uses signed chars. In the case of an 8 bit character, str2hashbuf() in lib/ext2fs/dirhash.c produces a different buf than the kernel, which leads to the problem that the hash calculated by TEA_transform() is a different one. Oh, dear. This is actually a kernel bug, because the on all other platforms, the TEA hash will be using a signed char --- and if you want filesystems to be portable between different systems (hint: we do), then all architectures should be using the same algorithm. And the vast majority of the systems out there are using signed chars. Unfortunately PowerPC decided to be different. :-( But given that e2fsck is doing it right, and the kernel is doing it wrong, we have the mismatch already. Sigh, this is going to be especially painful, given that all the major distributions (SLES, RHEL, Debian, Ubuntu, etc.) are now shipping with directory hashing enabled by default, and so this is going to impact a huge number of PowerPC Linux users and customers. Hmm, except isn't the problem ALREADY that PPC is broken with 8-bit chars and htree? That's what started this problem in the first place. Running e2fsck allowed the kernel htree code to find the file, when it could not otherwise be looked up... Need to verify that (my mental stack is overflowing). IIRC this problem was also reported in the past but no solution was found. I think fixing the kernel to specify signed chars for the hash will FIX the PPC kernel code. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc
On Fri, Sep 29, 2006 at 11:15:47AM -0600, Andreas Dilger wrote: Hmm, except isn't the problem ALREADY that PPC is broken with 8-bit chars and htree? That's what started this problem in the first place. Running e2fsck allowed the kernel htree code to find the file, when it could not otherwise be looked up... Need to verify that (my mental stack is overflowing). IIRC this problem was also reported in the past but no solution was found. I think fixing the kernel to specify signed chars for the hash will FIX the PPC kernel code. If the filesystem is empty (or at least no no hashtree directories), then when the kernel creates new directories and expands to the point where they become indexed, they will be indexed with the PPC variant of the hash algorithm. This will be self consistent, and everything will work fine --- until the filesystem gets corrupted to the point where e2fsprogs needs to rebuild one or more hashed directories. At that point the directories will be rebuilt using the same conventions used by all other conventions, but the directories will no longer be useful on the PPC kernel. Joerg, can you confirm this? On a PPC machine, can you create a smallish ext3 filesystem (say, 4-8 megabytes), create a directory with enough files in it that it becomes indexed (verify using lsattr), and show that it works just fine on a PPC. Now take that image, and transfer it to an x86 machine; you should find that the kernel can't look up any of the directories on the x86 machine. If you then run e2fsck -fD on that filesystem (running the e2fsck on either x86 or PPC; it shouldn't make a difference), then the resulting filesystem should work just fine on the x86, and fail on the PPC. If I understand this problem correctly, the above experiment should confirm what I suspect is going on. - Ted -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc
On Wed, Sep 27, 2006 at 02:22:51PM +0200, Jörg Sommer wrote: Package: e2fsprogs Version: 1.39-1 Severity: important you set the compiler option -fsigned-char, but on PowerPC the default is unsigned char. This makes the kernel uses unsigned and e2fsck uses signed chars. In the case of an 8 bit character, str2hashbuf() in lib/ext2fs/dirhash.c produces a different buf than the kernel, which leads to the problem that the hash calculated by TEA_transform() is a different one. Oh, dear. This is actually a kernel bug, because the on all other platforms, the TEA hash will be using a signed char --- and if you want filesystems to be portable between different systems (hint: we do), then all architectures should be using the same algorithm. And the vast majority of the systems out there are using signed chars. Unfortunately PowerPC decided to be different. :-( Fixing this is going to be painful, because if we fix the kernel to suddenly use signed chars in the TEA algorithm, like all other right thinking architectures (joke! joke!), then existing ext3 filesystems with directory hashing enabled will break. But given that e2fsck is doing it right, and the kernel is doing it wrong, we have the mismatch already. Sigh, this is going to be especially painful, given that all the major distributions (SLES, RHEL, Debian, Ubuntu, etc.) are now shipping with directory hashing enabled by default, and so this is going to impact a huge number of PowerPC Linux users and customers. OK, I think how are going to have to fix this is to define two new flags in the superblock, which indicates whether str2hashbuf is using signed or unsigned chars. If neither flag is set, then new kernels and new e2fsprogs will set the flag for the native signed/unsigned char convention, and assume that it is correct. I think this is only a PowerPC problem, but we'll probably need to do a quick check of all of the other architectures. Paul, since you're the PowerPC maintainer, I figured I'd better cc you. Does this sound sane to you? - Ted -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc
Package: e2fsprogs Version: 1.39-1 Severity: important Hi, you set the compiler option -fsigned-char, but on PowerPC the default is unsigned char. This makes the kernel uses unsigned and e2fsck uses signed chars. In the case of an 8 bit character, str2hashbuf() in lib/ext2fs/dirhash.c produces a different buf than the kernel, which leads to the problem that the hash calculated by TEA_transform() is a different one. If the hash is different enough from the hash calculated by the kernel, both search and insert the file in a different leave, which leads to the problem that files aren't accesable by the kernel if e2fsck has put them into the wrong leave. I have an small image where you can test it. With e2fsck from debian one file is not accessable, but with a fixed version (using unsigned char in str2hashbuf) the kernel can access the file. Are you interested in it? Bye, Jörg. -- System Information: Debian Release: unstable/experimental APT prefers unstable APT policy: (990, 'unstable'), (1, 'experimental') Architecture: powerpc (ppc) Shell: /bin/sh linked to /bin/dash Kernel: Linux 2.6.18 Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) Versions of packages e2fsprogs depends on: ii e2fslibs 1.39-1 ext2 filesystem libraries ii libblkid11.39-1 block device id library ii libc62.3.6.ds1-4 GNU C Library: Shared libraries ii libcomerr2 1.39-1 common error description library ii libdevmapper1.02 2:1.02.08-1 The Linux Kernel Device Mapper use ii libselinux1 1.30.28-1 SELinux shared libraries ii libsepol11.12.26-2 Security Enhanced Linux policy lib ii libss2 1.39-1 command-line interface parsing lib ii libuuid1 1.39-1 universally unique id library e2fsprogs recommends no packages. -- no debconf information pgp99G0fyAhJa.pgp Description: PGP signature
