Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
On Monday 05 January 2009 09:01:56 Jamie Lokier wrote: > Bernd Petrovitsch wrote: > > I assume that the NFS-mounted root filesystem is a real distribution. > > Not unless you call uClinux (MMU-less) a real distribution, no. I want things to be orthogonal. The following should be completely separate steps: 1) Creating a cross compiler 2) building a native development environment 3) booting a native development environment (on real hardware or under and emulator) 4) natively building your target system. You should be able to mix and match. Crosstool for #1, go download "fedora for arm" instead of #2, qemu or real hardware is your choice for #3, and then you should be able to natively build gentoo under an ubuntu host or vice versa. (How is not currently properly documented, but I'm working on that.) My objection to build systems like buildroot or uClinux is that they bundle all this together into a big hairball. They create their own cross compiler, build their own root file system, use their own packaging system, and you have to take it all or nothing. My build system is ruthlessly orthogonal. I try not to make it depend on other bits of _itself_ more than necessary. > > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > > proper shared libs. Feel free to fund this :-) > > > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > > insane IMHO. > > We have similar cross-build issues without MMUs... I.e. that a lot of > useful packages don't cross-build properly (including many which use > Autoconf), and it might be easier to make a native build environment > than to debug and patch all the broken-for-cross-build packages. > Especially as sometimes they build, but fail at run-time in some > conditions. If you can get a version of the same architecture with an mmu you can actually build natively on that. It's not ideal (it's a bit like trying to build i486 code on an i686; the fact it runs on the host is no guarantee it'll run on the target), but it's better than cross compiling. And most things have a broad enough compatible "base architecture" that you can mostly get away with it. > But you're right it's probably insane to try. I haven't dared as I > suspect GCC and/or Binutils would break too :-) Oh it does, but you can fix it. :) > I'm sticking instead with "oh well cross-build a few packages by hand > and just don't even _try_ to use most of the handy software out there". Cross compiling doesn't scale, and it bit-rots insanely quickly. > You mentioned ARM Debian. According to > http://wiki.debian.org/ArmEabiPort one recommended method of > bootstrapping it is building natively on an emulated ARM, because > cross-building is fragile. That's what my firmware linux project does too. (I believe I was one of the first doing this back in 2006, but there are three or four others out there doing it now.) Native compiling under emulation is an idea whose time has come. Emulators on cheap x86-64 laptops today are about as powerful as high end tricked out build servers circa 2001, and Moore's Law continues to advance. More memory, more CPU (maybe via SMP but distcc can take advantage of that today and qemu will develop threading someday). You can throw engineering time at the problem (making cross compiling work) or you can throw hardware at the problem (build natively and buy fast native or emulator-hosting hardware). The balance used to be in favor of the former; not so much anymore. That said, my drive for reproducibility and orthogonality says that your native development environment must be something you can reproduce entirely from source on an arbitrary host. You can't make cross compiling go away entirely, the best you can do is limit it to bootstrapping the native environment. But I want to keep the parts I have to cross compile as small and simple as possible, and then run a native build script to get a richer environment. For the past 5+ years my definition has been "an environment that can rebuild itself under itself is powerful enough, that's all I need to cross compile", and from the first time I tried this (late 2002) up until 2.6.25 that was 7 packages. That's why I responded to the addition of perl as a regression, because for my use case it was. > -- Jamie Rob -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
On Monday 05 January 2009 04:46:18 Bernd Petrovitsch wrote: > > My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted. > > Their /bin/sh does not do $((...)), and Bash is not there at all. > > I assume that the NFS-mounted root filesystem is a real distribution. > And on the local flash is a usual busybox based firmware. Building on an nfs mount is evil. Make cares greatly about timestamp accuracy, and NFS's dentry cacheing doesn't really, especially when it discards cached copies and re-fetches them, and the server and client's clocks are a half-second off from each other. Sometimes you haven't got a choice, but I hate having to debug the build problems this intermittently causes. If you never do anything except "make all" it should suck less. > > If I were installing GCC natively on them, I'd install GNU Make and a > > proper shell while I were at it. But I don't know if Bash works > > ACK. > > > properly without fork()* - or even if GCC does :-) > > > > Perl might be hard, as shared libraries aren't supported by the > > toolchain which targets my ARMs* and Perl likes its loadable modules. > > The simplest way to go is probably to use CentOS or Debian or another > ready binary distribution on ARM (or MIPS or PPC or whatever core the > embedded system has) possibly on a custom build kernel (if necessary). Building natively on target hardware or under QEMU is growing in popularity. That's how the non-x86 versions of major distros build, and they even have policy documents about it. Here's Fedora's: http://fedoraproject.org/wiki/Architectures/ARM#Native_Compilation And here are the guys who opened the door for Ubuntu's official Arm port: http://mojo.handhelds.org/files/HandheldsMojo_ELC2008.pdf Of course hobbyists like myself haven't got the budget to buy a cluster of high-end arm systems and they're not always even _available_ for things like cris, and for new architectures (Xylinx microblaze anyone?) you'll always have to cross compile to bootstrap the first development environment on 'em anyway, and it's nice for your environment to be _reproducible_... So a more flexible approach is to cross compile just enough to get a working native development environment on the target, and then continue the build natively (whether it's under qemu or on a sufficiently powerful piece of target hardware). That's what my "art piece" Firmware Linux project does, and there's a scratchbox rewrite (sbox2, http://www.freedesktop.org/wiki/Software/sbox2 ) that does the same sort of thing, and there are others out there in various states of development. With x86 hardware so cheap and powerful, building under emulation for less powerful targets starts to make sense. Building natively under emulation (QEMU) is available to hobbyists like me and avoids most of the fun cross compiling issues you don't find out about until after you've shipped the system and somebody tries to do something with it you didn't test. So far the record for diagnosing one of these is the two full- time weeks my friend Garrett spent back at TimeSys tracking down why perl signal handling wasn't working on mips; turned out it was using x86 signal numbers rather which don't match the mips ones. The BSP had been shipping for over a year at that point, but nobody had ever tried to do signal handling in perl on mips before, and since the perl ./configure step is written in perl finding the broken part took some doing. This was back in the mists of early 2007 so it's ancient history by now, of course... If you have set up a cross compiler, you can configure QEMU to use distcc to call out through its virtual network to the cross compiler running on the host, which gives you a speed boost without reintroducing most of the horrible cross compiling issues: there's still only a native toolchain so your build doesn't have to keep two contexts (hostcc/targetcc) straight, ./configure still runs natively so any binaries it builds can run and any questions it asks about the host it's building on should give the right answers for the target it's building for (including uname -m and friends), headers are #included natively and libraries are linked natively (that's just how distcc works, preprocessing and linking happen on the local machine) and there's only one set so they can't accidentally mix and the cross compiler isn't even _involved_ in that, make runs natively so it won't get confused by strange environment variables (yeah, seen that one)...) Only the heavy lifting of compiling preprocessed .c files to .o files gets exported, which is the one thing the cross compiler can't really screw up. But bootstraping a native build environment to run under the emulator is something you want to keep down to as few packages as possible, because if you're trying to get the same behavior across half a dozen boards, cross compiling breaks every time you upgrade _anything_. > [...] > > > (* - No MMU on some
Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
On Mon, 2009-01-05 at 15:01 +, Jamie Lokier wrote: > Bernd Petrovitsch wrote: > > I assume that the NFS-mounted root filesystem is a real distribution. > > Not unless you call uClinux (MMU-less) a real distribution, no. Not really. > > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > > proper shared libs. Feel free to fund this :-) > > > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > > insane IMHO. > > We have similar cross-build issues without MMUs... I.e. that a lot of Of course. > useful packages don't cross-build properly (including many which use > Autoconf), and it might be easier to make a native build environment Tell me about it - AC_TRY_RUN() is the culprit. And `pkg-config` supports cross-compilation only since 18 months or so. Before one had to rewrite the generated .pc files. [...] > You mentioned ARM Debian. According to > http://wiki.debian.org/ArmEabiPort one recommended method of > bootstrapping it is building natively on an emulated ARM, because > cross-building is fragile. That's of course the other solution - if qemu supports your $EMBEDDED_CPU good enough. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
Bernd Petrovitsch wrote: > I assume that the NFS-mounted root filesystem is a real distribution. Not unless you call uClinux (MMU-less) a real distribution, no. > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > > proper shared libs. Feel free to fund this :-) > > The above mentioned ARMs have a MMU. Without MMU, it would be truly > insane IMHO. We have similar cross-build issues without MMUs... I.e. that a lot of useful packages don't cross-build properly (including many which use Autoconf), and it might be easier to make a native build environment than to debug and patch all the broken-for-cross-build packages. Especially as sometimes they build, but fail at run-time in some conditions. But you're right it's probably insane to try. I haven't dared as I suspect GCC and/or Binutils would break too :-) I'm sticking instead with "oh well cross-build a few packages by hand and just don't even _try_ to use most of the handy software out there". You mentioned ARM Debian. According to http://wiki.debian.org/ArmEabiPort one recommended method of bootstrapping it is building natively on an emulated ARM, because cross-building is fragile. -- Jamie -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 12/17] Squashfs: header files
Hi Phillip,
On Mon, Jan 5, 2009 at 1:08 PM, Phillip Lougher
wrote:
> +#define TRACE(s, args...) pr_debug("SQUASHFS: "s, ## args)
You've probably heard this before but silly "tracing" such as:
TRACE("Entered squashfs_fill_superblock\n");
should really be removed from the filesystem code.
> +#define ERROR(s, args...) pr_err("SQUASHFS error: "s, ## args)
> +
> +#define WARNING(s, args...)pr_warning("SQUASHFS: "s, ## args)
I think you're supposed to #define pr_fmt() in your header instead of
adding wrappers like these.
Pekka
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 00/17] Squashfs: compressed read-only filesystem
Hi Phillip. On Mon, Jan 05, 2009 at 11:08:23AM +, Phillip Lougher (phil...@lougher.demon.co.uk) wrote: > This a second respin of the Squashfs patches incorporating the review comments > received. Thanks to everyone who have sent comments. > > Summary of changes in patch respin: > > 1. Vmalloc removed, smaller PAGE_CACHE_SIZE buffers are now allocated > 2. Renamed some global functions, prefixing with squashfs_ > 3. brelse changed to put_bh > 4. cache->lock coverage extended in squashfs_put_cache() and >squashfs_cache_get() > 5. New squashfs.txt file in Documentation/filesystems > 6. Changed 'long long' usage to u64 for variables referring to 64-bit >filesystem locations > 7. SQUASHFS_I() renamed to squashfs_i() > 8. Renamed locked variable to refcount to clarify usage > 9. Renamed waiting variable to num_waiters, making it clear it is a count >rather than a boolean > 10. Made pending and error fields int rather than char Looks good. You can also update the year in the copyright string in the files :) -- Evgeniy Polyakov -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 01/17] Squashfs: inode operations
Hi.
On Mon, Jan 05, 2009 at 11:08:23AM +, Phillip Lougher
(phil...@lougher.demon.co.uk) wrote:
> +int squashfs_read_inode(struct inode *inode, long long ino)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct squashfs_sb_info *msblk = sb->s_fs_info;
> + u64 block = SQUASHFS_INODE_BLK(ino) + msblk->inode_table;
> + int err, type, offset = SQUASHFS_INODE_OFFSET(ino);
> + union squashfs_inode squashfs_ino;
> + struct squashfs_base_inode *sqshb_ino = &squashfs_ino.base;
> +
What's the size of that union? If big enough it will lead to some
problems.
> + TRACE("Entered squashfs_read_inode\n");
> +
> + /*
> + * Read inode base common to all inode types.
> + */
> + err = squashfs_read_metadata(sb, sqshb_ino, &block,
> + &offset, sizeof(*sqshb_ino));
> + if (err < 0)
> + goto failed_read;
> +
> + err = squashfs_new_inode(sb, inode, sqshb_ino);
> + if (err)
> + goto failed_read;
> +
> + block = SQUASHFS_INODE_BLK(ino) + msblk->inode_table;
> + offset = SQUASHFS_INODE_OFFSET(ino);
> +
> + type = le16_to_cpu(sqshb_ino->inode_type);
> + switch (type) {
> + case SQUASHFS_REG_TYPE: {
> + unsigned int frag_offset, frag_size, frag;
> + u64 frag_blk;
Above variables can be moved out of the switch, since they are used in
some other cases too.
--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 02/17] Squashfs: directory lookup operations
Hi Phillip.
One possible 'show-stopper' below and couple trivials.
On Mon, Jan 05, 2009 at 11:08:23AM +, Phillip Lougher
(phil...@lougher.demon.co.uk) wrote:
> +static int get_dir_index_using_name(struct super_block *sb,
> + u64 *next_block, int *next_offset, u64 index_start,
> + int index_offset, int i_count, const char *name,
> + int len)
> +{
> + struct squashfs_sb_info *msblk = sb->s_fs_info;
> + int i, size, length = 0, err;
> + struct squashfs_dir_index *index;
> + char *str;
> +
> + TRACE("Entered get_dir_index_using_name, i_count %d\n", i_count);
> +
> + index = kmalloc(sizeof(*index) + SQUASHFS_NAME_LEN * 2 + 2, GFP_KERNEL);
> + if (index == NULL) {
> + ERROR("Failed to allocate squashfs_dir_index\n");
> + goto out;
> + }
> +
> + str = &index->name[SQUASHFS_NAME_LEN + 1];
> + strncpy(str, name, len);
> + str[len] = '\0';
> +
> + for (i = 0; i < i_count; i++) {
> + err = squashfs_read_metadata(sb, index, &index_start,
> + &index_offset, sizeof(*index));
> + if (err < 0)
> + break;
> +
> +
Double new line. Code-style purists will scream when see this.
This was a show-stopper.
> + size = le32_to_cpu(index->size) + 1;
> +
> + err = squashfs_read_metadata(sb, index->name, &index_start,
> + &index_offset, size);
> + if (err < 0)
> + break;
> +
> + index->name[size] = '\0';
> +
> + if (strcmp(index->name, str) > 0)
> + break;
> +
> + length = le32_to_cpu(index->index);
> + *next_block = le32_to_cpu(index->start_block) +
> + msblk->directory_table;
> + }
> +
> + *next_offset = (length + *next_offset) % SQUASHFS_METADATA_SIZE;
> + kfree(index);
> +
> +out:
> + /*
> + * Return index (f_pos) of the looked up metadata block. Translate
> + * from internal f_pos to external f_pos which is offset by 3 because
> + * we invent "." and ".." entries which are not actually stored in the
> + * directory.
> + */
> + return length + 3;
> +}
> +
> +
Another double new-line show-stopper.
> +static struct dentry *squashfs_lookup(struct inode *dir, struct dentry
> *dentry,
> + struct nameidata *nd)
> +{
> + const unsigned char *name = dentry->d_name.name;
> + int len = dentry->d_name.len;
> + struct inode *inode = NULL;
> + struct squashfs_sb_info *msblk = dir->i_sb->s_fs_info;
> + struct squashfs_dir_header dirh;
> + struct squashfs_dir_entry *dire;
> + u64 block = squashfs_i(dir)->start + msblk->directory_table;
> + int offset = squashfs_i(dir)->offset;
> + int err, length = 0, dir_count, size;
> +
> + TRACE("Entered squashfs_lookup [%llx:%x]\n", block, offset);
> +
> + dire = kmalloc(sizeof(*dire) + SQUASHFS_NAME_LEN + 1, GFP_KERNEL);
> + if (dire == NULL) {
> + ERROR("Failed to allocate squashfs_dir_entry\n");
> + return ERR_PTR(-ENOMEM);
> + }
> +
> + if (len > SQUASHFS_NAME_LEN) {
> + err = -ENAMETOOLONG;
> + goto failed;
> + }
> +
> + length = get_dir_index_using_name(dir->i_sb, &block, &offset,
> + squashfs_i(dir)->dir_idx_start,
> + squashfs_i(dir)->dir_idx_offset,
> + squashfs_i(dir)->dir_idx_cnt, name, len);
> +
You do not check the return value here.
Plus dir entry allocation can be done after above len check.
This one is trivial.
--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
On Son, 2009-01-04 at 22:50 -0600, Rob Landley wrote: > On Sunday 04 January 2009 18:15:30 Bernd Petrovitsch wrote: [...] > > ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat > > too extreme. > > I have yet to encounter a system that uses dash _without_ bash. (All ubuntu Hmm, should be doable with a chroot environment quite cheap and simple. > variants, even jeos, install bash by default. They moved the /bin/sh symlink Yes, I know (small) embedded systems that have a bash (and not "only" one of busybox shells). It eases writing somewhat fast shell scripts without the need for lots of fork()s+exec()s too . Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 11/17] Squashfs: block operations
Again, none of my comments are criticism against merging squashfs. Code
improvements should continue independently.
On Mon, 5 January 2009 11:08:24 +, Phillip Lougher wrote:
> +/*
> + * Read and decompress a metadata block or datablock. Length is non-zero
> + * if a datablock is being read (the size is stored elsewhere in the
> + * filesystem), otherwise the length is obtained from the first two bytes of
> + * the metadata block. A bit in the length field indicates if the block
> + * is stored uncompressed in the filesystem (usually because compression
> + * generated a larger block - this does occasionally happen with zlib).
> + */
This is the core function of squashfs. As such, it could probably use
some kerneldoc that explains each parameter and the return value;
> +int squashfs_read_data(struct super_block *sb, void **buffer, u64 index,
> + int length, u64 *next_index, int srclength)
> +{
> + struct squashfs_sb_info *msblk = sb->s_fs_info;
> + struct buffer_head **bh;
> + int offset = index & ((1 << msblk->devblksize_log2) - 1);
> + u64 cur_index = index >> msblk->devblksize_log2;
> + int bytes, compressed, b = 0, k = 0, page = 0, avail;
> +
> +
> + bh = kcalloc((msblk->block_size >> msblk->devblksize_log2) + 1,
> + sizeof(*bh), GFP_KERNEL);
> + if (bh == NULL)
> + return -ENOMEM;
> +
> + if (length) {
> + /*
> + * Datablock.
> + */
> + bytes = -offset;
> + compressed = SQUASHFS_COMPRESSED_BLOCK(length);
> + length = SQUASHFS_COMPRESSED_SIZE_BLOCK(length);
> + if (next_index)
> + *next_index = index + length;
> +
> + TRACE("Block @ 0x%llx, %scompressed size %d, src size %d\n",
> + index, compressed ? "" : "un", length, srclength);
> +
> + if (length < 0 || length > srclength ||
> + (index + length) > msblk->bytes_used)
> + goto read_failure;
> +
> + for (b = 0; bytes < length; b++, cur_index++) {
> + bh[b] = sb_getblk(sb, cur_index);
> + if (bh[b] == NULL)
> + goto block_release;
> + bytes += msblk->devblksize;
> + }
> + ll_rw_block(READ, b, bh);
> + } else {
> + /*
> + * Metadata block.
> + */
> + if ((index + 2) > msblk->bytes_used)
> + goto read_failure;
> +
> + bh[0] = get_block_length(sb, &cur_index, &offset, &length);
> + if (bh[0] == NULL)
> + goto read_failure;
> + b = 1;
> +
> + bytes = msblk->devblksize - offset;
> + compressed = SQUASHFS_COMPRESSED(length);
> + length = SQUASHFS_COMPRESSED_SIZE(length);
> + if (next_index)
> + *next_index = index + length + 2;
> +
> + TRACE("Block @ 0x%llx, %scompressed size %d\n", index,
> + compressed ? "" : "un", length);
> +
> + if (length < 0 || length > srclength ||
> + (index + length) > msblk->bytes_used)
> + goto block_release;
> +
> + for (; bytes < length; b++) {
> + bh[b] = sb_getblk(sb, ++cur_index);
> + if (bh[b] == NULL)
> + goto block_release;
> + bytes += msblk->devblksize;
> + }
> + ll_rw_block(READ, b - 1, bh + 1);
> + }
> +
> + if (compressed) {
This looks like a prime candidate for a seperate function.
> + int zlib_err = 0, zlib_init = 0;
> +
> + /*
> + * Uncompress block.
> + */
> +
> + mutex_lock(&msblk->read_data_mutex);
> +
> + msblk->stream.avail_out = 0;
> + msblk->stream.avail_in = 0;
> +
> + bytes = length;
> + do {
> + if (msblk->stream.avail_in == 0 && k < b) {
> + avail = min(bytes, msblk->devblksize - offset);
> + bytes -= avail;
> + wait_on_buffer(bh[k]);
> + if (!buffer_uptodate(bh[k]))
> + goto release_mutex;
> +
> + if (avail == 0) {
> + offset = 0;
> + put_bh(bh[k++]);
> + continue;
> + }
> +
> + msblk->stream.next_in = bh[k]->b_data + offset;
> + msblk->stream.avail_in = avail;
> + offset = 0;
> + }
> +
> + if (msblk
[PATCH V3 06/17] Squashfs: super block operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/super.c | 440 +++
1 files changed, 440 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
new file mode 100644
index 000..a0466d7
--- /dev/null
+++ b/fs/squashfs/super.c
@@ -0,0 +1,440 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * super.c
+ */
+
+/*
+ * This file implements code to read the superblock, read and initialise
+ * in-memory structures at mount time, and all the VFS glue code to register
+ * the filesystem.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+static struct file_system_type squashfs_fs_type;
+static struct super_operations squashfs_super_ops;
+
+static int supported_squashfs_filesystem(short major, short minor, short comp)
+{
+ if (major < SQUASHFS_MAJOR) {
+ ERROR("Major/Minor mismatch, older Squashfs %d.%d "
+ "filesystems are unsupported\n", major, minor);
+ return -EINVAL;
+ } else if (major > SQUASHFS_MAJOR || minor > SQUASHFS_MINOR) {
+ ERROR("Major/Minor mismatch, trying to mount newer "
+ "%d.%d filesystem\n", major, minor);
+ ERROR("Please update your kernel\n");
+ return -EINVAL;
+ }
+
+ if (comp != ZLIB_COMPRESSION)
+ return -EINVAL;
+
+ return 0;
+}
+
+
+static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
+{
+ struct squashfs_sb_info *msblk;
+ struct squashfs_super_block *sblk = NULL;
+ char b[BDEVNAME_SIZE];
+ struct inode *root;
+ long long root_inode;
+ unsigned short flags;
+ unsigned int fragments;
+ u64 lookup_table_start;
+ int err;
+
+ TRACE("Entered squashfs_fill_superblock\n");
+
+ sb->s_fs_info = kzalloc(sizeof(*msblk), GFP_KERNEL);
+ if (sb->s_fs_info == NULL) {
+ ERROR("Failed to allocate squashfs_sb_info\n");
+ return -ENOMEM;
+ }
+ msblk = sb->s_fs_info;
+
+ msblk->stream.workspace = kmalloc(zlib_inflate_workspacesize(),
+ GFP_KERNEL);
+ if (msblk->stream.workspace == NULL) {
+ ERROR("Failed to allocate zlib workspace\n");
+ goto failure;
+ }
+
+ sblk = kzalloc(sizeof(*sblk), GFP_KERNEL);
+ if (sblk == NULL) {
+ ERROR("Failed to allocate squashfs_super_block\n");
+ goto failure;
+ }
+
+ msblk->devblksize = sb_min_blocksize(sb, BLOCK_SIZE);
+ msblk->devblksize_log2 = ffz(~msblk->devblksize);
+
+ mutex_init(&msblk->read_data_mutex);
+ mutex_init(&msblk->meta_index_mutex);
+
+ /*
+* msblk->bytes_used is checked in squashfs_read_table to ensure reads
+* are not beyond filesystem end. But as we're using
+* squashfs_read_table here to read the superblock (including the value
+* of bytes_used) we need to set it to an initial sensible dummy value
+*/
+ msblk->bytes_used = sizeof(*sblk);
+ err = squashfs_read_table(sb, sblk, SQUASHFS_START, sizeof(*sblk));
+
+ if (err < 0) {
+ ERROR("unable to read squashfs_super_block\n");
+ goto failed_mount;
+ }
+
+ /* Check it is a SQUASHFS superblock */
+ sb->s_magic = le32_to_cpu(sblk->s_magic);
+ if (sb->s_magic != SQUASHFS_MAGIC) {
+ if (!silent)
+ ERROR("Can't find a SQUASHFS superblock on %s\n",
+ bdevname(sb->s_bdev, b));
+ err = -EINVAL;
+ goto failed_mount;
+ }
+
+ /* Check the MAJOR & MINOR versions and compression type */
+ err = supported_squashfs_filesystem(le16_to_cpu(sblk->s_major),
+ le16_to_cpu(sblk->s_minor),
+ le16_to_cpu(sblk->compression));
+ if (err < 0)
+ goto failed_mount;
+
+
[PATCH V3 09/17] Squashfs: uid/gid lookup operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/id.c | 94 ++
1 files changed, 94 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/id.c b/fs/squashfs/id.c
new file mode 100644
index 000..3795b83
--- /dev/null
+++ b/fs/squashfs/id.c
@@ -0,0 +1,94 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * id.c
+ */
+
+/*
+ * This file implements code to handle uids and gids.
+ *
+ * For space efficiency regular files store uid and gid indexes, which are
+ * converted to 32-bit uids/gids using an id look up table. This table is
+ * stored compressed into metadata blocks. A second index table is used to
+ * locate these. This second index table for speed of access (and because it
+ * is small) is read at mount time and cached in memory.
+ */
+
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Map uid/gid index into real 32-bit uid/gid using the id look up table
+ */
+int squashfs_get_id(struct super_block *sb, unsigned int index,
+ unsigned int *id)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ int block = SQUASHFS_ID_BLOCK(index);
+ int offset = SQUASHFS_ID_BLOCK_OFFSET(index);
+ u64 start_block = le64_to_cpu(msblk->id_table[block]);
+ __le32 disk_id;
+ int err;
+
+ err = squashfs_read_metadata(sb, &disk_id, &start_block, &offset,
+ sizeof(disk_id));
+ if (err < 0)
+ return err;
+
+ *id = le32_to_cpu(disk_id);
+ return 0;
+}
+
+
+/*
+ * Read uncompressed id lookup table indexes from disk into memory
+ */
+__le64 *squashfs_read_id_index_table(struct super_block *sb,
+ u64 id_table_start, unsigned short no_ids)
+{
+ unsigned int length = SQUASHFS_ID_BLOCK_BYTES(no_ids);
+ __le64 *id_table;
+ int err;
+
+ TRACE("In read_id_index_table, length %d\n", length);
+
+ /* Allocate id lookup table indexes */
+ id_table = kmalloc(length, GFP_KERNEL);
+ if (id_table == NULL) {
+ ERROR("Failed to allocate id index table\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ err = squashfs_read_table(sb, id_table, id_table_start, length);
+ if (err < 0) {
+ ERROR("unable to read id index table\n");
+ kfree(id_table);
+ return ERR_PTR(err);
+ }
+
+ return id_table;
+}
--
1.5.6.3
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 01/17] Squashfs: inode operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/inode.c | 346 +++
1 files changed, 346 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/inode.c b/fs/squashfs/inode.c
new file mode 100644
index 000..7a63398
--- /dev/null
+++ b/fs/squashfs/inode.c
@@ -0,0 +1,346 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * inode.c
+ */
+
+/*
+ * This file implements code to create and read inodes from disk.
+ *
+ * Inodes in Squashfs are identified by a 48-bit inode which encodes the
+ * location of the compressed metadata block containing the inode, and the byte
+ * offset into that block where the inode is placed ().
+ *
+ * To maximise compression there are different inodes for each file type
+ * (regular file, directory, device, etc.), the inode contents and length
+ * varying with the type.
+ *
+ * To further maximise compression, two types of regular file inode and
+ * directory inode are defined: inodes optimised for frequently occurring
+ * regular files and directories, and extended types where extra
+ * information has to be stored.
+ */
+
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Initialise VFS inode with the base inode information common to all
+ * Squashfs inode types. Sqsh_ino contains the unswapped base inode
+ * off disk.
+ */
+static int squashfs_new_inode(struct super_block *sb, struct inode *inode,
+ struct squashfs_base_inode *sqsh_ino)
+{
+ int err;
+
+ err = squashfs_get_id(sb, le16_to_cpu(sqsh_ino->uid), &inode->i_uid);
+ if (err)
+ return err;
+
+ err = squashfs_get_id(sb, le16_to_cpu(sqsh_ino->guid), &inode->i_gid);
+ if (err)
+ return err;
+
+ inode->i_ino = le32_to_cpu(sqsh_ino->inode_number);
+ inode->i_mtime.tv_sec = le32_to_cpu(sqsh_ino->mtime);
+ inode->i_atime.tv_sec = inode->i_mtime.tv_sec;
+ inode->i_ctime.tv_sec = inode->i_mtime.tv_sec;
+ inode->i_mode = le16_to_cpu(sqsh_ino->mode);
+ inode->i_size = 0;
+
+ return err;
+}
+
+
+struct inode *squashfs_iget(struct super_block *sb, long long ino,
+ unsigned int ino_number)
+{
+ struct inode *inode = iget_locked(sb, ino_number);
+ int err;
+
+ TRACE("Entered squashfs_iget\n");
+
+ if (!inode)
+ return ERR_PTR(-ENOMEM);
+ if (!(inode->i_state & I_NEW))
+ return inode;
+
+ err = squashfs_read_inode(inode, ino);
+ if (err) {
+ iget_failed(inode);
+ return ERR_PTR(err);
+ }
+
+ unlock_new_inode(inode);
+ return inode;
+}
+
+
+/*
+ * Initialise VFS inode by reading inode from inode table (compressed
+ * metadata). The format and amount of data read depends on type.
+ */
+int squashfs_read_inode(struct inode *inode, long long ino)
+{
+ struct super_block *sb = inode->i_sb;
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ u64 block = SQUASHFS_INODE_BLK(ino) + msblk->inode_table;
+ int err, type, offset = SQUASHFS_INODE_OFFSET(ino);
+ union squashfs_inode squashfs_ino;
+ struct squashfs_base_inode *sqshb_ino = &squashfs_ino.base;
+
+ TRACE("Entered squashfs_read_inode\n");
+
+ /*
+* Read inode base common to all inode types.
+*/
+ err = squashfs_read_metadata(sb, sqshb_ino, &block,
+ &offset, sizeof(*sqshb_ino));
+ if (err < 0)
+ goto failed_read;
+
+ err = squashfs_new_inode(sb, inode, sqshb_ino);
+ if (err)
+ goto failed_read;
+
+ block = SQUASHFS_INODE_BLK(ino) + msblk->inode_table;
+ offset = SQUASHFS_INODE_OFFSET(ino);
+
+ type = le16_to_cpu(sqshb_ino->inode_type);
+ switch (type) {
+ case SQUASHFS_REG_TYPE: {
+ unsigned int frag_offset, frag_size, frag;
+ u64 frag_blk;
+ struct squashfs_reg_inode *sqsh_ino = &squashfs_ino.reg;
+
+ err = squashfs_read_metadata(sb, s
[PATCH V3 10/17] Squashfs: cache operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/cache.c | 412 +++
1 files changed, 412 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c
new file mode 100644
index 000..f29eda1
--- /dev/null
+++ b/fs/squashfs/cache.c
@@ -0,0 +1,412 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * cache.c
+ */
+
+/*
+ * Blocks in Squashfs are compressed. To avoid repeatedly decompressing
+ * recently accessed data Squashfs uses two small metadata and fragment caches.
+ *
+ * This file implements a generic cache implementation used for both caches,
+ * plus functions layered ontop of the generic cache implementation to
+ * access the metadata and fragment caches.
+ *
+ * To avoid out of memory and fragmentation isssues with vmalloc the cache
+ * uses sequences of kmalloced PAGE_CACHE_SIZE buffers.
+ *
+ * It should be noted that the cache is not used for file datablocks, these
+ * are decompressed and cached in the page-cache in the normal way. The
+ * cache is only used to temporarily cache fragment and metadata blocks
+ * which have been read as as a result of a metadata (i.e. inode or
+ * directory) or fragment access. Because metadata and fragments are packed
+ * together into blocks (to gain greater compression) the read of a particular
+ * piece of metadata or fragment will retrieve other metadata/fragments which
+ * have been packed with it, these because of locality-of-reference may be read
+ * in the near future. Temporarily caching them ensures they are available for
+ * near future access without requiring an additional read and decompress.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Look-up block in cache, and increment usage count. If not in cache, read
+ * and decompress it from disk.
+ */
+struct squashfs_cache_entry *squashfs_cache_get(struct super_block *sb,
+ struct squashfs_cache *cache, u64 block, int length)
+{
+ int i, n;
+ struct squashfs_cache_entry *entry;
+
+ spin_lock(&cache->lock);
+
+ while (1) {
+ for (i = 0; i < cache->entries; i++)
+ if (cache->entry[i].block == block)
+ break;
+
+ if (i == cache->entries) {
+ /*
+* Block not in cache, if all cache entries are used
+* go to sleep waiting for one to become available.
+*/
+ if (cache->unused == 0) {
+ cache->num_waiters++;
+ spin_unlock(&cache->lock);
+ wait_event(cache->wait_queue, cache->unused);
+ spin_lock(&cache->lock);
+ cache->num_waiters--;
+ continue;
+ }
+
+ /*
+* At least one unused cache entry. A simple
+* round-robin strategy is used to choose the entry to
+* be evicted from the cache.
+*/
+ i = cache->next_blk;
+ for (n = 0; n < cache->entries; n++) {
+ if (cache->entry[i].refcount == 0)
+ break;
+ i = (i + 1) % cache->entries;
+ }
+
+ cache->next_blk = (i + 1) % cache->entries;
+ entry = &cache->entry[i];
+
+ /*
+* Initialise choosen cache entry, and fill it in from
+* disk.
+*/
+ cache->unused--;
+ entry->block = block;
+ entry->refcount = 1;
+ entry->pending = 1;
+ entry->num_waiters = 0;
+
[PATCH V3 16/17] Squashfs: documentation
Signed-off-by: Phillip Lougher --- Documentation/filesystems/squashfs.txt | 225 1 files changed, 225 insertions(+), 0 deletions(-) diff --git a/Documentation/filesystems/squashfs.txt b/Documentation/filesystems/squashfs.txt new file mode 100644 index 000..3e79e4a --- /dev/null +++ b/Documentation/filesystems/squashfs.txt @@ -0,0 +1,225 @@ +SQUASHFS 4.0 FILESYSTEM +=== + +Squashfs is a compressed read-only filesystem for Linux. +It uses zlib compression to compress files, inodes and directories. +Inodes in the system are very small and all blocks are packed to minimise +data overhead. Block sizes greater than 4K are supported up to a maximum +of 1Mbytes (default block size 128K). + +Squashfs is intended for general read-only filesystem use, for archival +use (i.e. in cases where a .tar.gz file may be used), and in constrained +block device/memory systems (e.g. embedded systems) where low overhead is +needed. + +Mailing list: squashfs-de...@lists.sourceforge.net +Web site: www.squashfs.org + +1. FILESYSTEM FEATURES +-- + +Squashfs filesystem features versus Cramfs: + + SquashfsCramfs + +Max filesystem size: 2^6416 MiB +Max file size: ~ 2 TiB 16 MiB +Max files: unlimited unlimited +Max directories: unlimited unlimited +Max entries per directory: unlimited unlimited +Max block size:1 MiB 4 KiB +Metadata compression: yes no +Directory indexes: yes no +Sparse file support: yes no +Tail-end packing (fragments): yes no +Exportable (NFS etc.): yes no +Hard link support: yes no +"." and ".." in readdir: yes no +Real inode numbers:yes no +32-bit uids/gids: yes no +File creation time:yes no +Xattr and ACL support: no no + +Squashfs compresses data, inodes and directories. In addition, inode and +directory data are highly compacted, and packed on byte boundaries. Each +compressed inode is on average 8 bytes in length (the exact length varies on +file type, i.e. regular file, directory, symbolic link, and block/char device +inodes have different sizes). + +2. USING SQUASHFS +- + +As squashfs is a read-only filesystem, the mksquashfs program must be used to +create populated squashfs filesystems. This and other squashfs utilities +can be obtained from http://www.squashfs.org. Usage instructions can be +obtained from this site also. + + +3. SQUASHFS FILESYSTEM DESIGN +- + +A squashfs filesystem consists of seven parts, packed together on a byte +alignment: + +--- + | superblock | + |---| + | datablocks | + | & fragments | + |---| + | inode table | + |---| + | directory | + | table | + |---| + | fragment| + |table | + |---| + |export | + |table | + |---| + |uid/gid| + | lookup table | +--- + +Compressed data blocks are written to the filesystem as files are read from +the source directory, and checked for duplicates. Once all file data has been +written the completed inode, directory, fragment, export and uid/gid lookup +tables are written. + +3.1 Inodes +-- + +Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each +compressed block is prefixed by a two byte length, the top bit is set if the +block is uncompressed. A block will be uncompressed if the -noI option is set, +or if the compressed block was larger than the uncompressed block. + +Inodes are packed into the metadata blocks, and are not aligned to block +boundaries, therefore inodes overlap compressed blocks. Inodes are identified +by a 48-bit number which encodes the location of the compressed metadata block +containing the inode, and the byte offset into that block where the inode is +placed (). + +To maximise compression there are different inodes for each file type +(regular file, directory, device, etc.), the inode contents and length +varying with the type. + +To further maximise compression, two types of regular file inode and +directory inode are defined: inodes optimised for frequently occurring +regular files and directories, and extended types where extra +information has to be stored. + +3.2 Directories +--- + +Like inodes, directories are packed into compressed
[PATCH V3 17/17] MAINTAINERS: squashfs entry
Signed-off-by: Phillip Lougher --- MAINTAINERS |7 +++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index befacf0..6ed506f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4049,6 +4049,13 @@ L: cbe-oss-...@ozlabs.org W: http://www.ibm.com/developerworks/power/cell/ S: Supported +SQUASHFS FILE SYSTEM +P: Phillip Lougher +M: phil...@lougher.demon.co.uk +L: squashfs-de...@lists.sourceforge.net (subscribers-only) +W: http://squashfs.org.uk +S: Maintained + SRM (Alpha) environment access P: Jan-Benedict Glaw M: jbg...@lug-owl.de -- 1.5.6.3 -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 07/17] Squashfs: export operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/export.c | 155 ++
1 files changed, 155 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/export.c b/fs/squashfs/export.c
new file mode 100644
index 000..69e971d
--- /dev/null
+++ b/fs/squashfs/export.c
@@ -0,0 +1,155 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * export.c
+ */
+
+/*
+ * This file implements code to make Squashfs filesystems exportable (NFS etc.)
+ *
+ * The export code uses an inode lookup table to map inode numbers passed in
+ * filehandles to an inode location on disk. This table is stored compressed
+ * into metadata blocks. A second index table is used to locate these. This
+ * second index table for speed of access (and because it is small) is read at
+ * mount time and cached in memory.
+ *
+ * The inode lookup table is used only by the export code, inode disk
+ * locations are directly encoded in directories, enabling direct access
+ * without an intermediate lookup for all operations except the export ops.
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Look-up inode number (ino) in table, returning the inode location.
+ */
+static long long squashfs_inode_lookup(struct super_block *sb, int ino_num)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ int blk = SQUASHFS_LOOKUP_BLOCK(ino_num - 1);
+ int offset = SQUASHFS_LOOKUP_BLOCK_OFFSET(ino_num - 1);
+ u64 start = le64_to_cpu(msblk->inode_lookup_table[blk]);
+ __le64 ino;
+ int err;
+
+ TRACE("Entered squashfs_inode_lookup, inode_number = %d\n", ino_num);
+
+ err = squashfs_read_metadata(sb, &ino, &start, &offset, sizeof(ino));
+ if (err < 0)
+ return err;
+
+ TRACE("squashfs_inode_lookup, inode = 0x%llx\n",
+ (u64) le64_to_cpu(ino));
+
+ return le64_to_cpu(ino);
+}
+
+
+static struct dentry *squashfs_export_iget(struct super_block *sb,
+ unsigned int ino_num)
+{
+ long long ino;
+ struct dentry *dentry = ERR_PTR(-ENOENT);
+
+ TRACE("Entered squashfs_export_iget\n");
+
+ ino = squashfs_inode_lookup(sb, ino_num);
+ if (ino >= 0)
+ dentry = d_obtain_alias(squashfs_iget(sb, ino, ino_num));
+
+ return dentry;
+}
+
+
+static struct dentry *squashfs_fh_to_dentry(struct super_block *sb,
+ struct fid *fid, int fh_len, int fh_type)
+{
+ if ((fh_type != FILEID_INO32_GEN && fh_type != FILEID_INO32_GEN_PARENT)
+ || fh_len < 2)
+ return NULL;
+
+ return squashfs_export_iget(sb, fid->i32.ino);
+}
+
+
+static struct dentry *squashfs_fh_to_parent(struct super_block *sb,
+ struct fid *fid, int fh_len, int fh_type)
+{
+ if (fh_type != FILEID_INO32_GEN_PARENT || fh_len < 4)
+ return NULL;
+
+ return squashfs_export_iget(sb, fid->i32.parent_ino);
+}
+
+
+static struct dentry *squashfs_get_parent(struct dentry *child)
+{
+ struct inode *inode = child->d_inode;
+ unsigned int parent_ino = squashfs_i(inode)->parent;
+
+ return squashfs_export_iget(inode->i_sb, parent_ino);
+}
+
+
+/*
+ * Read uncompressed inode lookup table indexes off disk into memory
+ */
+__le64 *squashfs_read_inode_lookup_table(struct super_block *sb,
+ u64 lookup_table_start, unsigned int inodes)
+{
+ unsigned int length = SQUASHFS_LOOKUP_BLOCK_BYTES(inodes);
+ __le64 *inode_lookup_table;
+ int err;
+
+ TRACE("In read_inode_lookup_table, length %d\n", length);
+
+ /* Allocate inode lookup table indexes */
+ inode_lookup_table = kmalloc(length, GFP_KERNEL);
+ if (inode_lookup_table == NULL) {
+ ERROR("Failed to allocate inode lookup table\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ err = squashfs_read_table(sb, inode_lookup_table, lookup_table_start,
+ length);
+ if (err < 0) {
+ ERROR("unable to read inode lookup table\n");
+
[PATCH V3 04/17] Squashfs: regular file operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/file.c | 502
1 files changed, 502 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/file.c b/fs/squashfs/file.c
new file mode 100644
index 000..717767d
--- /dev/null
+++ b/fs/squashfs/file.c
@@ -0,0 +1,502 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * file.c
+ */
+
+/*
+ * This file contains code for handling regular files. A regular file
+ * consists of a sequence of contiguous compressed blocks, and/or a
+ * compressed fragment block (tail-end packed block). The compressed size
+ * of each datablock is stored in a block list contained within the
+ * file inode (itself stored in one or more compressed metadata blocks).
+ *
+ * To speed up access to datablocks when reading 'large' files (256 Mbytes or
+ * larger), the code implements an index cache that caches the mapping from
+ * block index to datablock location on disk.
+ *
+ * The index cache allows Squashfs to handle large files (up to 1.75 TiB) while
+ * retaining a simple and space-efficient block list on disk. The cache
+ * is split into slots, caching up to eight 224 GiB files (128 KiB blocks).
+ * Larger files use multiple slots, with 1.75 TiB files using all 8 slots.
+ * The index cache is designed to be memory efficient, and by default uses
+ * 16 KiB.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Locate cache slot in range [offset, index] for specified inode. If
+ * there's more than one return the slot closest to index.
+ */
+static struct meta_index *locate_meta_index(struct inode *inode, int offset,
+ int index)
+{
+ struct meta_index *meta = NULL;
+ struct squashfs_sb_info *msblk = inode->i_sb->s_fs_info;
+ int i;
+
+ mutex_lock(&msblk->meta_index_mutex);
+
+ TRACE("locate_meta_index: index %d, offset %d\n", index, offset);
+
+ if (msblk->meta_index == NULL)
+ goto not_allocated;
+
+ for (i = 0; i < SQUASHFS_META_SLOTS; i++) {
+ if (msblk->meta_index[i].inode_number == inode->i_ino &&
+ msblk->meta_index[i].offset >= offset &&
+ msblk->meta_index[i].offset <= index &&
+ msblk->meta_index[i].locked == 0) {
+ TRACE("locate_meta_index: entry %d, offset %d\n", i,
+ msblk->meta_index[i].offset);
+ meta = &msblk->meta_index[i];
+ offset = meta->offset;
+ }
+ }
+
+ if (meta)
+ meta->locked = 1;
+
+not_allocated:
+ mutex_unlock(&msblk->meta_index_mutex);
+
+ return meta;
+}
+
+
+/*
+ * Find and initialise an empty cache slot for index offset.
+ */
+static struct meta_index *empty_meta_index(struct inode *inode, int offset,
+ int skip)
+{
+ struct squashfs_sb_info *msblk = inode->i_sb->s_fs_info;
+ struct meta_index *meta = NULL;
+ int i;
+
+ mutex_lock(&msblk->meta_index_mutex);
+
+ TRACE("empty_meta_index: offset %d, skip %d\n", offset, skip);
+
+ if (msblk->meta_index == NULL) {
+ /*
+* First time cache index has been used, allocate and
+* initialise. The cache index could be allocated at
+* mount time but doing it here means it is allocated only
+* if a 'large' file is read.
+*/
+ msblk->meta_index = kcalloc(SQUASHFS_META_SLOTS,
+ sizeof(*(msblk->meta_index)), GFP_KERNEL);
+ if (msblk->meta_index == NULL) {
+ ERROR("Failed to allocate meta_index\n");
+ goto failed;
+ }
+ for (i = 0; i < SQUASHFS_META_SLOTS; i++) {
+ msblk->meta_index[i].inode_number = 0;
+ msblk->meta_index[i].locked = 0;
+
[PATCH V3 11/17] Squashfs: block operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/block.c | 274 +++
1 files changed, 274 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
new file mode 100644
index 000..c837dfc
--- /dev/null
+++ b/fs/squashfs/block.c
@@ -0,0 +1,274 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * block.c
+ */
+
+/*
+ * This file implements the low-level routines to read and decompress
+ * datablocks and metadata blocks.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Read the metadata block length, this is stored in the first two
+ * bytes of the metadata block.
+ */
+static struct buffer_head *get_block_length(struct super_block *sb,
+ u64 *cur_index, int *offset, int *length)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ struct buffer_head *bh;
+
+ bh = sb_bread(sb, *cur_index);
+ if (bh == NULL)
+ return NULL;
+
+ if (msblk->devblksize - *offset == 1) {
+ *length = (unsigned char) bh->b_data[*offset];
+ put_bh(bh);
+ bh = sb_bread(sb, ++(*cur_index));
+ if (bh == NULL)
+ return NULL;
+ *length |= (unsigned char) bh->b_data[0] << 8;
+ *offset = 1;
+ } else {
+ *length = (unsigned char) bh->b_data[*offset] |
+ (unsigned char) bh->b_data[*offset + 1] << 8;
+ *offset += 2;
+ }
+
+ return bh;
+}
+
+
+/*
+ * Read and decompress a metadata block or datablock. Length is non-zero
+ * if a datablock is being read (the size is stored elsewhere in the
+ * filesystem), otherwise the length is obtained from the first two bytes of
+ * the metadata block. A bit in the length field indicates if the block
+ * is stored uncompressed in the filesystem (usually because compression
+ * generated a larger block - this does occasionally happen with zlib).
+ */
+int squashfs_read_data(struct super_block *sb, void **buffer, u64 index,
+ int length, u64 *next_index, int srclength)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ struct buffer_head **bh;
+ int offset = index & ((1 << msblk->devblksize_log2) - 1);
+ u64 cur_index = index >> msblk->devblksize_log2;
+ int bytes, compressed, b = 0, k = 0, page = 0, avail;
+
+
+ bh = kcalloc((msblk->block_size >> msblk->devblksize_log2) + 1,
+ sizeof(*bh), GFP_KERNEL);
+ if (bh == NULL)
+ return -ENOMEM;
+
+ if (length) {
+ /*
+* Datablock.
+*/
+ bytes = -offset;
+ compressed = SQUASHFS_COMPRESSED_BLOCK(length);
+ length = SQUASHFS_COMPRESSED_SIZE_BLOCK(length);
+ if (next_index)
+ *next_index = index + length;
+
+ TRACE("Block @ 0x%llx, %scompressed size %d, src size %d\n",
+ index, compressed ? "" : "un", length, srclength);
+
+ if (length < 0 || length > srclength ||
+ (index + length) > msblk->bytes_used)
+ goto read_failure;
+
+ for (b = 0; bytes < length; b++, cur_index++) {
+ bh[b] = sb_getblk(sb, cur_index);
+ if (bh[b] == NULL)
+ goto block_release;
+ bytes += msblk->devblksize;
+ }
+ ll_rw_block(READ, b, bh);
+ } else {
+ /*
+* Metadata block.
+*/
+ if ((index + 2) > msblk->bytes_used)
+ goto read_failure;
+
+ bh[0] = get_block_length(sb, &cur_index, &offset, &length);
+ if (bh[0] == NULL)
+ goto read_failure;
+ b = 1;
+
+ bytes = msblk->devblksize - offset;
+
[PATCH V3 12/17] Squashfs: header files
Signed-off-by: Phillip Lougher
---
fs/squashfs/squashfs.h | 90 ++
fs/squashfs/squashfs_fs.h| 381 ++
fs/squashfs/squashfs_fs_i.h | 45 +
fs/squashfs/squashfs_fs_sb.h | 76 +
4 files changed, 592 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
new file mode 100644
index 000..6b2515d
--- /dev/null
+++ b/fs/squashfs/squashfs.h
@@ -0,0 +1,90 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * squashfs.h
+ */
+
+#define TRACE(s, args...) pr_debug("SQUASHFS: "s, ## args)
+
+#define ERROR(s, args...) pr_err("SQUASHFS error: "s, ## args)
+
+#define WARNING(s, args...)pr_warning("SQUASHFS: "s, ## args)
+
+static inline struct squashfs_inode_info *squashfs_i(struct inode *inode)
+{
+ return list_entry(inode, struct squashfs_inode_info, vfs_inode);
+}
+
+/* block.c */
+extern int squashfs_read_data(struct super_block *, void **, u64, int, u64 *,
+ int);
+
+/* cache.c */
+extern struct squashfs_cache *squashfs_cache_init(char *, int, int);
+extern void squashfs_cache_delete(struct squashfs_cache *);
+extern struct squashfs_cache_entry *squashfs_cache_get(struct super_block *,
+ struct squashfs_cache *, u64, int);
+extern void squashfs_cache_put(struct squashfs_cache_entry *);
+extern int squashfs_copy_data(void *, struct squashfs_cache_entry *, int, int);
+extern int squashfs_read_metadata(struct super_block *, void *, u64 *,
+ int *, int);
+extern struct squashfs_cache_entry *squashfs_get_fragment(struct super_block *,
+ u64, int);
+extern struct squashfs_cache_entry *squashfs_get_datablock(struct super_block
*,
+ u64, int);
+extern int squashfs_read_table(struct super_block *, void *, u64, int);
+
+/* export.c */
+extern __le64 *squashfs_read_inode_lookup_table(struct super_block *, u64,
+ unsigned int);
+
+/* fragment.c */
+extern int squashfs_frag_lookup(struct super_block *, unsigned int, u64 *);
+extern __le64 *squashfs_read_fragment_index_table(struct super_block *,
+ u64, unsigned int);
+
+/* id.c */
+extern int squashfs_get_id(struct super_block *, unsigned int, unsigned int *);
+extern __le64 *squashfs_read_id_index_table(struct super_block *, u64,
+ unsigned short);
+
+/* inode.c */
+extern struct inode *squashfs_iget(struct super_block *, long long,
+ unsigned int);
+extern int squashfs_read_inode(struct inode *, long long);
+
+/*
+ * Inodes and files operations
+ */
+
+/* dir.c */
+extern const struct file_operations squashfs_dir_ops;
+
+/* export.c */
+extern const struct export_operations squashfs_export_ops;
+
+/* file.c */
+extern const struct address_space_operations squashfs_aops;
+
+/* namei.c */
+extern const struct inode_operations squashfs_dir_inode_ops;
+
+/* symlink.c */
+extern const struct address_space_operations squashfs_symlink_aops;
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
new file mode 100644
index 000..6840da1
--- /dev/null
+++ b/fs/squashfs/squashfs_fs.h
@@ -0,0 +1,381 @@
+#ifndef SQUASHFS_FS
+#define SQUASHFS_FS
+/*
+ * Squashfs
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * squashfs_fs.h
+ */
[PATCH V3 14/17] Squashfs: Kconfig entry
Signed-off-by: Phillip Lougher --- fs/Kconfig | 52 1 files changed, 52 insertions(+), 0 deletions(-) diff --git a/fs/Kconfig b/fs/Kconfig index ff0e819..2553e0b 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -894,6 +894,58 @@ config CRAMFS If unsure, say N. +config SQUASHFS + tristate "SquashFS 4.0 - Squashed file system support" + depends on BLOCK + select ZLIB_INFLATE + help + Saying Y here includes support for SquashFS 4.0 (a Compressed + Read-Only File System). Squashfs is a highly compressed read-only + filesystem for Linux. It uses zlib compression to compress both + files, inodes and directories. Inodes in the system are very small + and all blocks are packed to minimise data overhead. Block sizes + greater than 4K are supported up to a maximum of 1 Mbytes (default + block size 128K). SquashFS 4.0 supports 64 bit filesystems and files + (larger than 4GB), full uid/gid information, hard links and + timestamps. + + Squashfs is intended for general read-only filesystem use, for + archival use (i.e. in cases where a .tar.gz file may be used), and in + embedded systems where low overhead is needed. Further information + and tools are available from http://squashfs.sourceforge.net. + + If you want to compile this as a module ( = code which can be + inserted in and removed from the running kernel whenever you want), + say M here and read . The module + will be called squashfs. Note that the root file system (the one + containing the directory /) cannot be compiled as a module. + + If unsure, say N. + +config SQUASHFS_EMBEDDED + + bool "Additional option for memory-constrained systems" + depends on SQUASHFS + default n + help + Saying Y here allows you to specify cache size. + + If unsure, say N. + +config SQUASHFS_FRAGMENT_CACHE_SIZE + int "Number of fragments cached" if SQUASHFS_EMBEDDED + depends on SQUASHFS + default "3" + help + By default SquashFS caches the last 3 fragments read from + the filesystem. Increasing this amount may mean SquashFS + has to re-read fragments less often from disk, at the expense + of extra system memory. Decreasing this amount will mean + SquashFS uses less memory at the expense of extra reads from disk. + + Note there must be at least one cached fragment. Anything + much more than three will probably not make much difference. + config VXFS_FS tristate "FreeVxFS file system support (VERITAS VxFS(TM) compatible)" depends on BLOCK -- 1.5.6.3 -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 15/17] Squashfs: initrd support
Signed-off-by: Phillip Lougher
---
init/do_mounts_rd.c | 14 ++
1 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/init/do_mounts_rd.c b/init/do_mounts_rd.c
index a7c748f..0f0f0cf 100644
--- a/init/do_mounts_rd.c
+++ b/init/do_mounts_rd.c
@@ -9,6 +9,7 @@
#include
#include "do_mounts.h"
+#include "../fs/squashfs/squashfs_fs.h"
int __initdata rd_prompt = 1;/* 1 = prompt for RAM disk, 0 = don't prompt */
@@ -41,6 +42,7 @@ static int __init crd_load(int in_fd, int out_fd);
* ext2
* romfs
* cramfs
+ * squashfs
* gzip
*/
static int __init
@@ -51,6 +53,7 @@ identify_ramdisk_image(int fd, int start_block)
struct ext2_super_block *ext2sb;
struct romfs_super_block *romfsb;
struct cramfs_super *cramfsb;
+ struct squashfs_super_block *squashfsb;
int nblocks = -1;
unsigned char *buf;
@@ -62,6 +65,7 @@ identify_ramdisk_image(int fd, int start_block)
ext2sb = (struct ext2_super_block *) buf;
romfsb = (struct romfs_super_block *) buf;
cramfsb = (struct cramfs_super *) buf;
+ squashfsb = (struct squashfs_super_block *) buf;
memset(buf, 0xe5, size);
/*
@@ -99,6 +103,16 @@ identify_ramdisk_image(int fd, int start_block)
goto done;
}
+ /* squashfs is at block zero too */
+ if (le32_to_cpu(squashfsb->s_magic) == SQUASHFS_MAGIC) {
+ printk(KERN_NOTICE
+ "RAMDISK: squashfs filesystem found at block %d\n",
+ start_block);
+ nblocks = (le64_to_cpu(squashfsb->bytes_used) + BLOCK_SIZE - 1)
+>> BLOCK_SIZE_BITS;
+ goto done;
+ }
+
/*
* Read block 1 to test for minix and ext2 superblock
*/
--
1.5.6.3
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 02/17] Squashfs: directory lookup operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/namei.c | 242 +++
1 files changed, 242 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/namei.c b/fs/squashfs/namei.c
new file mode 100644
index 000..9e39865
--- /dev/null
+++ b/fs/squashfs/namei.c
@@ -0,0 +1,242 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * namei.c
+ */
+
+/*
+ * This file implements code to do filename lookup in directories.
+ *
+ * Like inodes, directories are packed into compressed metadata blocks, stored
+ * in a directory table. Directories are accessed using the start address of
+ * the metablock containing the directory and the offset into the
+ * decompressed block ().
+ *
+ * Directories are organised in a slightly complex way, and are not simply
+ * a list of file names. The organisation takes advantage of the
+ * fact that (in most cases) the inodes of the files will be in the same
+ * compressed metadata block, and therefore, can share the start block.
+ * Directories are therefore organised in a two level list, a directory
+ * header containing the shared start block value, and a sequence of directory
+ * entries, each of which share the shared start block. A new directory header
+ * is written once/if the inode start block changes. The directory
+ * header/directory entry list is repeated as many times as necessary.
+ *
+ * Directories are sorted, and can contain a directory index to speed up
+ * file lookup. Directory indexes store one entry per metablock, each entry
+ * storing the index/filename mapping to the first directory header
+ * in each metadata block. Directories are sorted in alphabetical order,
+ * and at lookup the index is scanned linearly looking for the first filename
+ * alphabetically larger than the filename being looked up. At this point the
+ * location of the metadata block the filename is in has been found.
+ * The general idea of the index is ensure only one metadata block needs to be
+ * decompressed to do a lookup irrespective of the length of the directory.
+ * This scheme has the advantage that it doesn't require extra memory overhead
+ * and doesn't require much extra storage on disk.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Lookup name in the directory index, returning the location of the metadata
+ * block containing it, and the directory index this represents.
+ *
+ * If we get an error reading the index then return the part of the index
+ * (if any) we have managed to read - the index isn't essential, just
+ * quicker.
+ */
+static int get_dir_index_using_name(struct super_block *sb,
+ u64 *next_block, int *next_offset, u64 index_start,
+ int index_offset, int i_count, const char *name,
+ int len)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ int i, size, length = 0, err;
+ struct squashfs_dir_index *index;
+ char *str;
+
+ TRACE("Entered get_dir_index_using_name, i_count %d\n", i_count);
+
+ index = kmalloc(sizeof(*index) + SQUASHFS_NAME_LEN * 2 + 2, GFP_KERNEL);
+ if (index == NULL) {
+ ERROR("Failed to allocate squashfs_dir_index\n");
+ goto out;
+ }
+
+ str = &index->name[SQUASHFS_NAME_LEN + 1];
+ strncpy(str, name, len);
+ str[len] = '\0';
+
+ for (i = 0; i < i_count; i++) {
+ err = squashfs_read_metadata(sb, index, &index_start,
+ &index_offset, sizeof(*index));
+ if (err < 0)
+ break;
+
+
+ size = le32_to_cpu(index->size) + 1;
+
+ err = squashfs_read_metadata(sb, index->name, &index_start,
+ &index_offset, size);
+ if (err < 0)
+ break;
+
+ index->name[size] = '\0';
+
+ if (strcmp(index->name, str) > 0)
+ break;
+
+ length = le32_to_cpu(ind
[PATCH V3 00/17] Squashfs: compressed read-only filesystem
Hi, This a second respin of the Squashfs patches incorporating the review comments received. Thanks to everyone who have sent comments. Summary of changes in patch respin: 1. Vmalloc removed, smaller PAGE_CACHE_SIZE buffers are now allocated 2. Renamed some global functions, prefixing with squashfs_ 3. brelse changed to put_bh 4. cache->lock coverage extended in squashfs_put_cache() and squashfs_cache_get() 5. New squashfs.txt file in Documentation/filesystems 6. Changed 'long long' usage to u64 for variables referring to 64-bit filesystem locations 7. SQUASHFS_I() renamed to squashfs_i() 8. Renamed locked variable to refcount to clarify usage 9. Renamed waiting variable to num_waiters, making it clear it is a count rather than a boolean 10. Made pending and error fields int rather than char These 17 patches are against the latest linux-2.6.git (will apply to 2.6.28). These patches are also available from git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git As requested by Andrew Morton I have prepared a tree for linux-next inclusion. This is git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-next.git The Squashfs patches have been tested against both linus-2.6.git and linux-next.git. Finally, I would like to acknowledge the financial support of the Consumer Embedded Linux Forum (CELF). They've made it possible for me to spend the last six months working full time on this mainlining attempt. Thanks Phillip -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 13/17] Squashfs: Makefiles
Signed-off-by: Phillip Lougher --- fs/Makefile |1 + fs/squashfs/Makefile |8 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/fs/Makefile b/fs/Makefile index e6f423d..3f8843c 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -73,6 +73,7 @@ obj-$(CONFIG_JBD) += jbd/ obj-$(CONFIG_JBD2) += jbd2/ obj-$(CONFIG_EXT2_FS) += ext2/ obj-$(CONFIG_CRAMFS) += cramfs/ +obj-$(CONFIG_SQUASHFS) += squashfs/ obj-y += ramfs/ obj-$(CONFIG_HUGETLBFS)+= hugetlbfs/ obj-$(CONFIG_CODA_FS) += coda/ diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile new file mode 100644 index 000..8258cf9 --- /dev/null +++ b/fs/squashfs/Makefile @@ -0,0 +1,8 @@ +# +# Makefile for the linux squashfs routines. +# + +obj-$(CONFIG_SQUASHFS) += squashfs.o +squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o +squashfs-y += namei.o super.o symlink.o +#squashfs-y += squashfs2_0.o -- 1.5.6.3 -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 03/17] Squashfs: directory readdir operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/dir.c | 235 +
1 files changed, 235 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/dir.c b/fs/squashfs/dir.c
new file mode 100644
index 000..566b0ea
--- /dev/null
+++ b/fs/squashfs/dir.c
@@ -0,0 +1,235 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * dir.c
+ */
+
+/*
+ * This file implements code to read directories from disk.
+ *
+ * See namei.c for a description of directory organisation on disk.
+ */
+
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+static const unsigned char squashfs_filetype_table[] = {
+ DT_UNKNOWN, DT_DIR, DT_REG, DT_LNK, DT_BLK, DT_CHR, DT_FIFO, DT_SOCK
+};
+
+/*
+ * Lookup offset (f_pos) in the directory index, returning the
+ * metadata block containing it.
+ *
+ * If we get an error reading the index then return the part of the index
+ * (if any) we have managed to read - the index isn't essential, just
+ * quicker.
+ */
+static int get_dir_index_using_offset(struct super_block *sb,
+ u64 *next_block, int *next_offset, u64 index_start, int index_offset,
+ int i_count, u64 f_pos)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ int err, i, index, length = 0;
+ struct squashfs_dir_index dir_index;
+
+ TRACE("Entered get_dir_index_using_offset, i_count %d, f_pos %lld\n",
+ i_count, f_pos);
+
+ /*
+* Translate from external f_pos to the internal f_pos. This
+* is offset by 3 because we invent "." and ".." entries which are
+* not actually stored in the directory.
+*/
+ if (f_pos < 3)
+ return f_pos;
+ f_pos -= 3;
+
+ for (i = 0; i < i_count; i++) {
+ err = squashfs_read_metadata(sb, &dir_index, &index_start,
+ &index_offset, sizeof(dir_index));
+ if (err < 0)
+ break;
+
+ index = le32_to_cpu(dir_index.index);
+ if (index > f_pos)
+ /*
+* Found the index we're looking for.
+*/
+ break;
+
+ err = squashfs_read_metadata(sb, NULL, &index_start,
+ &index_offset, le32_to_cpu(dir_index.size) + 1);
+ if (err < 0)
+ break;
+
+ length = index;
+ *next_block = le32_to_cpu(dir_index.start_block) +
+ msblk->directory_table;
+ }
+
+ *next_offset = (length + *next_offset) % SQUASHFS_METADATA_SIZE;
+
+ /*
+* Translate back from internal f_pos to external f_pos.
+*/
+ return length + 3;
+}
+
+
+static int squashfs_readdir(struct file *file, void *dirent, filldir_t filldir)
+{
+ struct inode *inode = file->f_dentry->d_inode;
+ struct squashfs_sb_info *msblk = inode->i_sb->s_fs_info;
+ u64 block = squashfs_i(inode)->start + msblk->directory_table;
+ int offset = squashfs_i(inode)->offset, length = 0, dir_count, size,
+ type, err;
+ unsigned int inode_number;
+ struct squashfs_dir_header dirh;
+ struct squashfs_dir_entry *dire;
+
+ TRACE("Entered squashfs_readdir [%llx:%x]\n", block, offset);
+
+ dire = kmalloc(sizeof(*dire) + SQUASHFS_NAME_LEN + 1, GFP_KERNEL);
+ if (dire == NULL) {
+ ERROR("Failed to allocate squashfs_dir_entry\n");
+ goto finish;
+ }
+
+ /*
+* Return "." and ".." entries as the first two filenames in the
+* directory. To maximise compression these two entries are not
+* stored in the directory, and so we invent them here.
+*
+* It also means that the external f_pos is offset by 3 from the
+* on-disk directory f_pos.
+*/
+ while (file->f_pos < 3) {
+ char *name;
+ int i_ino;
+
+ if (file->f_pos
[PATCH V3 05/17] Squashfs: symlink operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/symlink.c | 118 +
1 files changed, 118 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/symlink.c b/fs/squashfs/symlink.c
new file mode 100644
index 000..83d8788
--- /dev/null
+++ b/fs/squashfs/symlink.c
@@ -0,0 +1,118 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * symlink.c
+ */
+
+/*
+ * This file implements code to handle symbolic links.
+ *
+ * The data contents of symbolic links are stored inside the symbolic
+ * link inode within the inode table. This allows the normally small symbolic
+ * link to be compressed as part of the inode table, achieving much greater
+ * compression than if the symbolic link was compressed individually.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+static int squashfs_symlink_readpage(struct file *file, struct page *page)
+{
+ struct inode *inode = page->mapping->host;
+ struct super_block *sb = inode->i_sb;
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ int index = page->index << PAGE_CACHE_SHIFT;
+ u64 block = squashfs_i(inode)->start;
+ int offset = squashfs_i(inode)->offset;
+ int length = min_t(int, i_size_read(inode) - index, PAGE_CACHE_SIZE);
+ int bytes, copied;
+ void *pageaddr;
+ struct squashfs_cache_entry *entry;
+
+ TRACE("Entered squashfs_symlink_readpage, page index %ld, start block "
+ "%llx, offset %x\n", page->index, block, offset);
+
+ /*
+* Skip index bytes into symlink metadata.
+*/
+ if (index) {
+ bytes = squashfs_read_metadata(sb, NULL, &block, &offset,
+ index);
+ if (bytes < 0) {
+ ERROR("Unable to read symlink [%llx:%x]\n",
+ squashfs_i(inode)->start,
+ squashfs_i(inode)->offset);
+ goto error_out;
+ }
+ }
+
+ /*
+* Read length bytes from symlink metadata. Squashfs_read_metadata
+* is not used here because it can sleep and we want to use
+* kmap_atomic to map the page. Instead call the underlying
+* squashfs_cache_get routine. As length bytes may overlap metadata
+* blocks, we may need to call squashfs_cache_get multiple times.
+*/
+ for (bytes = 0; bytes < length; offset = 0, bytes += copied) {
+ entry = squashfs_cache_get(sb, msblk->block_cache, block, 0);
+ if (entry->error) {
+ ERROR("Unable to read symlink [%llx:%x]\n",
+ squashfs_i(inode)->start,
+ squashfs_i(inode)->offset);
+ squashfs_cache_put(entry);
+ goto error_out;
+ }
+
+ pageaddr = kmap_atomic(page, KM_USER0);
+ copied = squashfs_copy_data(pageaddr + bytes, entry, offset,
+ length - bytes);
+ if (copied == length - bytes)
+ memset(pageaddr + length, 0, PAGE_CACHE_SIZE - length);
+ else
+ block = entry->next_index;
+ kunmap_atomic(pageaddr, KM_USER0);
+ squashfs_cache_put(entry);
+ }
+
+ flush_dcache_page(page);
+ SetPageUptodate(page);
+ unlock_page(page);
+ return 0;
+
+error_out:
+ SetPageError(page);
+ unlock_page(page);
+ return 0;
+}
+
+
+const struct address_space_operations squashfs_symlink_aops = {
+ .readpage = squashfs_symlink_readpage
+};
--
1.5.6.3
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 08/17] Squashfs: fragment block operations
Signed-off-by: Phillip Lougher
---
fs/squashfs/fragment.c | 98
1 files changed, 98 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/fragment.c b/fs/squashfs/fragment.c
new file mode 100644
index 000..b5a2c15
--- /dev/null
+++ b/fs/squashfs/fragment.c
@@ -0,0 +1,98 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * fragment.c
+ */
+
+/*
+ * This file implements code to handle compressed fragments (tail-end packed
+ * datablocks).
+ *
+ * Regular files contain a fragment index which is mapped to a fragment
+ * location on disk and compressed size using a fragment lookup table.
+ * Like everything in Squashfs this fragment lookup table is itself stored
+ * compressed into metadata blocks. A second index table is used to locate
+ * these. This second index table for speed of access (and because it
+ * is small) is read at mount time and cached in memory.
+ */
+
+#include
+#include
+#include
+#include
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+
+/*
+ * Look-up fragment using the fragment index table. Return the on disk
+ * location of the fragment and its compressed size
+ */
+int squashfs_frag_lookup(struct super_block *sb, unsigned int fragment,
+ u64 *fragment_block)
+{
+ struct squashfs_sb_info *msblk = sb->s_fs_info;
+ int block = SQUASHFS_FRAGMENT_INDEX(fragment);
+ int offset = SQUASHFS_FRAGMENT_INDEX_OFFSET(fragment);
+ u64 start_block = le64_to_cpu(msblk->fragment_index[block]);
+ struct squashfs_fragment_entry fragment_entry;
+ int size;
+
+ size = squashfs_read_metadata(sb, &fragment_entry, &start_block,
+ &offset, sizeof(fragment_entry));
+ if (size < 0)
+ return size;
+
+ *fragment_block = le64_to_cpu(fragment_entry.start_block);
+ size = le32_to_cpu(fragment_entry.size);
+
+ return size;
+}
+
+
+/*
+ * Read the uncompressed fragment lookup table indexes off disk into memory
+ */
+__le64 *squashfs_read_fragment_index_table(struct super_block *sb,
+ u64 fragment_table_start, unsigned int fragments)
+{
+ unsigned int length = SQUASHFS_FRAGMENT_INDEX_BYTES(fragments);
+ __le64 *fragment_index;
+ int err;
+
+ /* Allocate fragment lookup table indexes */
+ fragment_index = kmalloc(length, GFP_KERNEL);
+ if (fragment_index == NULL) {
+ ERROR("Failed to allocate fragment index table\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ err = squashfs_read_table(sb, fragment_index, fragment_table_start,
+ length);
+ if (err < 0) {
+ ERROR("unable to read fragment index table\n");
+ kfree(fragment_index);
+ return ERR_PTR(err);
+ }
+
+ return fragment_index;
+}
--
1.5.6.3
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
On Mon, 2009-01-05 at 02:23 +, Jamie Lokier wrote: > Bernd Petrovitsch wrote: > > > (I have 850 Linux boxes on my network with a bourne shell which > > > doesn't do $((...)). I won't be building kernels on them though :-) > > > > Believe it or not, but there are folks out there who build the firmware > > on ARM 200 MHz NFS-mounted systems natively (and not simply > > cross-compile it on a 2GHz PC .). > > Really? > > My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted. > Their /bin/sh does not do $((...)), and Bash is not there at all. I assume that the NFS-mounted root filesystem is a real distribution. And on the local flash is a usual busybox based firmware. > If I were installing GCC natively on them, I'd install GNU Make and a > proper shell while I were at it. But I don't know if Bash works ACK. > properly without fork()* - or even if GCC does :-) > > Perl might be hard, as shared libraries aren't supported by the > toolchain which targets my ARMs* and Perl likes its loadable modules. The simplest way to go is probably to use CentOS or Debian or another ready binary distribution on ARM (or MIPS or PPC or whatever core the embedded system has) possibly on a custom build kernel (if necessary). [...] > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add > proper shared libs. Feel free to fund this :-) The above mentioned ARMs have a MMU. Without MMU, it would be truly insane IMHO. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
