Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hello, On Tue, 23 Jun 2009 21:40:34 +0200 Jo-Philipp Wich wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hi Frédéric. > > Thanks for trying it out. > The Fonera2 uses "kernel" as partition name too, so I had a chance to > test it. > > Attached is a newer version of platform.sh that does some more sanity > checking on the mtd partitions and a simple patch for the OpenWrt > buildroot (8.09 branch) to easily generate the required images. Works perfectly. The sysupgrade-atheros package in the LuCI feed should depend on base-files-atheros instead of base-files, opkg does not find the required dependency as it is. > > The naming could be better and the stuff is in no way meant to interfere > with the real solution worked on by Felix, it's just a nice hack to have > sysupgrade now. I understand very well, and it is a very useful hack :) Regards fred ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi Frédéric.
Thanks for trying it out.
The Fonera2 uses "kernel" as partition name too, so I had a chance to
test it.
Attached is a newer version of platform.sh that does some more sanity
checking on the mtd partitions and a simple patch for the OpenWrt
buildroot (8.09 branch) to easily generate the required images.
The naming could be better and the stuff is in no way meant to interfere
with the real solution worked on by Felix, it's just a nice hack to have
sysupgrade now.
Regards,
JoW
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkpBL7IACgkQdputYINPTPOORwCcCK+a92K6ZHz01bjDkM5nRyEF
kmQAoIv5BQLYlrQyjy6rWQ5X0SCBGdr5
=0nn6
-END PGP SIGNATURE-
platform.sh
Description: Bourne shell script
Index: target/linux/atheros/image/Makefile
===
--- target/linux/atheros/image/Makefile (revision 16280)
+++ target/linux/atheros/image/Makefile (working copy)
@@ -41,6 +41,17 @@
-r $(BIN_DIR)/openwrt-$(BOARD)-root.$(1) \
-o $(BIN_DIR)/openwrt-$(BOARD)-ubnt2-pico2-$(1).bin
+ ifeq ($(BOARD),atheros)
+ -sh $(TOPDIR)/scripts/atheros-combined-image.sh \
+ "$(BIN_DIR)/openwrt-$(BOARD)-vmlinux.lzma" "vmlinux.bin.l7" \
+ "$(BIN_DIR)/openwrt-$(BOARD)-root.$(1)" "rootfs" \
+ "$(BIN_DIR)/openwrt-fonera-combined.img"
+
+ -sh $(TOPDIR)/scripts/atheros-combined-image.sh \
+ "$(BIN_DIR)/openwrt-$(BOARD)-vmlinux.lzma" "kernel" \
+ "$(BIN_DIR)/openwrt-$(BOARD)-root.$(1)" "rootfs" \
+ "$(BIN_DIR)/openwrt-ubiquity-combined.img"
+ endif
endef
$(eval $(call BuildImage))
Index: scripts/atheros-combined-image.sh
===
--- scripts/atheros-combined-image.sh (revision 0)
+++ scripts/atheros-combined-image.sh (revision 0)
@@ -0,0 +1,12 @@
+#!/bin/sh
+
+[ -f "$1" -a -n "$2" -a -f "$3" -a -n "$4" ] || {
+ echo "Usage: $0 /path/to/kernel.image kernel_name /path/to/rootfs.image rootfs_name [output-file]"
+ exit 1
+}
+
+( printf "FF%08x%-16s%08x%-16s%65486s" \
+ $(stat -c "%s" "$1") ${2:-vmlinux.bin.l7} \
+ $(stat -c "%s" "$3") ${4:-rootfs} "";
+ cat "$1" "$3"
+) > ${5:-openwrt-atheros-combined.img}
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hello, On Sat, 20 Jun 2009 20:39:05 +0200 Jo-Philipp Wich wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hi Malte. > > > There's no reason why the unified format couldn't be a tgz (or some other, > > maybe uncompressed, format which carries a magic number) with some meta > > informations :) Daniel's proposal sounded quite complicated to me. > > So I finally took some time to try the .tgz approach on a classic > Fonera. As it turned out, the space in /tmp is not enough to store both > the tgz and the unpacked images at the same time. > > To solve that I basically "invented" some simple image format with a two > byte magic value in the beginning, followed by a 8 byte hex sequence > descriping the length of the kernel image, a 16 byte space-padded string > containing the name of the kernel mtd partition, a 8 byte hex-string > containing the length of the rootfs image and a space-padded 16 byte > string containing the name of the rootfs partition. the following 65486 > bytes are space-padded to make the initial "header" stuff 64KB aligned. > > This initial 64KB block is concatenated together with the kernel lzma > image and the rootfs image. The reason for doing such an awkward format > is, that I wanted to be able to extract the sub-images with busybox' dd > utility. > > Second step was to implement a platform.sh backend for sysupgrade which > uses "dd" to extract name and length information from the first 64KB of > the combined image. After determining the length information, "dd" is > used again to extract the sub-images from the correct offsets in the > combined image. The output is piped to the mtd utility. > > The configuration keeping is implemented with mtd's -j option when > writing the rootfs. So far it seemed to work but I have only tested > refleshing with the same images, need to try images of different size later. > > > I've attached my platform.sh script and another script to generate a > suitable combined image. I have no idea whether they will work with some > other device than the Fonera, maybe someone want to try this out. It works for me on a Nanostation5. I attach again the mkimg.sh script with the possibility to override partition names. Since the kernel partition name is "kernel" for Nanostations, I used : KERNEL_PART=kernel ./mkimg.sh openwrt-atheros-vmlinux.lzma openwrt-atheros-root.squashfs I tested it also with an image with different rootfs size, it seems ok. I think it will work with flash partitions as long as the new kernel is not one block bigger than the old one. If one wants to flash a bigger kernel he has to get rid of the dependency on the bootloader partition table information or be able to edit it (like Ubiquity appears to do with fwupdate). I think Felix's plan is welcome :) Thank you for the solution. fred mkimg.sh Description: application/shellscript ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
On Wed, 17 Jun 2009 23:08:17 +0200 Felix Fietkau wrote: > My plan for the unified format is to keep the on-flash format the same > as the input format. In the long run, I don't want any unpacking to be > done in sysupgrade or the web interface. The structure is roughly > like this: [second stage loader][environment][kernel][rootfs] Ok, here's my question. The brcm-63xx *require* a Broadcom format imagetag at the start of the flash image (on flash and for CFE upgrades), and won't load without it. Do you mean to keep this imagetag and have the [second stage loader][environment][kernel][rootfs] all in the kernel partition (as far as the CFE is concerned; for the OpenWRT obviously the partitions would be determined by the environment) ? Also, how does the second stage loader compare to the kernel? I ask because I have run across a router for which the CFE verifies the CRC of the kernel after it has been decompressed. Would the unified format be a problem for this router? Also, when it comes to decompression, do the 63xx routers have the CFE do the decompression or is it a loader? I ask because I know that the kernel has to be compressed with specific lzma parameters for it to work with the CFE and that indicates to me that it's not an arbitrary loader that's doing the unpacking, but rather the CFE. Or do you run the loader *after* un-lzma by the CFE? -- And that's my crabbing done for the day. Got it out of the way early, now I have the rest of the afternoon to sniff fragrant tea-roses or strangle cute bunnies or something. -- Michael Devore GnuPG Key Fingerprint 86 F5 81 A5 D4 2E 1F 1C http://gnupg.org The C Shore (Daniel Dickinson's Website) http://www.bmts.com/~cshore signature.asc Description: PGP signature ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Malte. > There's no reason why the unified format couldn't be a tgz (or some other, > maybe uncompressed, format which carries a magic number) with some meta > informations :) Daniel's proposal sounded quite complicated to me. So I finally took some time to try the .tgz approach on a classic Fonera. As it turned out, the space in /tmp is not enough to store both the tgz and the unpacked images at the same time. To solve that I basically "invented" some simple image format with a two byte magic value in the beginning, followed by a 8 byte hex sequence descriping the length of the kernel image, a 16 byte space-padded string containing the name of the kernel mtd partition, a 8 byte hex-string containing the length of the rootfs image and a space-padded 16 byte string containing the name of the rootfs partition. the following 65486 bytes are space-padded to make the initial "header" stuff 64KB aligned. This initial 64KB block is concatenated together with the kernel lzma image and the rootfs image. The reason for doing such an awkward format is, that I wanted to be able to extract the sub-images with busybox' dd utility. Second step was to implement a platform.sh backend for sysupgrade which uses "dd" to extract name and length information from the first 64KB of the combined image. After determining the length information, "dd" is used again to extract the sub-images from the correct offsets in the combined image. The output is piped to the mtd utility. The configuration keeping is implemented with mtd's -j option when writing the rootfs. So far it seemed to work but I have only tested refleshing with the same images, need to try images of different size later. I've attached my platform.sh script and another script to generate a suitable combined image. I have no idea whether they will work with some other device than the Fonera, maybe someone want to try this out. Regards, JoW -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAko9LMMACgkQdputYINPTPOcCgCfWeLtMYS9VhMmb3bacXaJPzPn 9IsAn190+7UVtxyoOpK2+uSF94lVXJB1 =K16A -END PGP SIGNATURE- mkimg.sh Description: Bourne shell script platform.sh Description: Bourne shell script ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Malte S. Stretz wrote: > Hi JoW, > > On Wednesday 17 June 2009 16:14:37 Jo-Philipp Wich wrote: >> I like that .tar.gz idea, it would also cover devices like the Fonera >> where you have two separate images too. >> >> However, it's planned to introduce a unified image format for OpenWrt >> which would solve most of this issues so I haven't started working on it. > > Is this planned for some time or do you mean Daniel's proposal (which sounds > like a new idea but I'm not subscribed to this list very long). > > There's no reason why the unified format couldn't be a tgz (or some other, > maybe uncompressed, format which carries a magic number) with some meta > informations :) Daniel's proposal sounded quite complicated to me. My plan for the unified format is to keep the on-flash format the same as the input format. In the long run, I don't want any unpacking to be done in sysupgrade or the web interface. The structure is roughly like this: [second stage loader][environment][kernel][rootfs] The second stage loader unpacks the kernel part using lzma, and the environment contains information about the split between kernel and rootfs, so that it no longer has to be reflected in the boot loader's partition split. Another useful aspect of the environment is that it can contain preloaded uci config changes for the image, so you can build preconfigured images for lots of nodes without repacking the kernel or the rootfs. I don't know when I'll be able to finish this, but I did start working on the loader part already (which is the biggest part of this) - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi, On Wednesday 17 June 2009 22:01:24 Peter Korsgaard wrote: > Malte> A broken tgz header will be recognized by these tools already. > > Malte> If we need any logic for flashing, we can also introduce a > Malte> sysupgrade dir. Anything in there will go to /lib/upgrade > Malte> before flashing. For even more flexibility, the root could > Malte> contain an optional (executable) sysupgrade.sh which called > Malte> instead of the system sysupgrade. > > Why not just use a standard format like the Debian .deb instead of > inventing a custom format? [...] Alternatively you could use ipkg/opkg. Indeed, I think this is an even better idea. M -- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
> "Malte" == Malte S Stretz writes: Hi, Malte> A broken tgz header will be recognized by these tools already. Malte> If we need any logic for flashing, we can also introduce a Malte> sysupgrade dir. Anything in there will go to /lib/upgrade Malte> before flashing. For even more flexibility, the root could Malte> contain an optional (executable) sysupgrade.sh which called Malte> instead of the system sysupgrade. Why not just use a standard format like the Debian .deb instead of inventing a custom format? A Debian .deb is nothing more than a tar.gz with some meta data (version number/description/dependencies/conflicts) which can be used to ensure you can only install the upgrade on compatible hw, and 4 scripts: - preinst, which runs before installation - posting, which runs after installation - prerm, which runs before removal (/upgrade) - postrm, which runs after removal (/upgrade) Busybox contains a dpkg implementation. I'm using such a setup, where the .deb will extract into something like /tmp/upgrade and the postinst script will copy the individual files to the correct mtd partitions. Alternatively you could use ipkg/opkg. -- Bye, Peter Korsgaard ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi JoW, On Wednesday 17 June 2009 16:14:37 Jo-Philipp Wich wrote: > I like that .tar.gz idea, it would also cover devices like the Fonera > where you have two separate images too. > > However, it's planned to introduce a unified image format for OpenWrt > which would solve most of this issues so I haven't started working on it. Is this planned for some time or do you mean Daniel's proposal (which sounds like a new idea but I'm not subscribed to this list very long). There's no reason why the unified format couldn't be a tgz (or some other, maybe uncompressed, format which carries a magic number) with some meta informations :) Daniel's proposal sounded quite complicated to me. > But imho it would be great to have such a capability for the stable 8.09 > series since it can be added as package on top of the normal code. > > The question is whether config-keeping is doable in such a way, have you > already tried this (with mtd's -j option) ? No, I haven't. The RB411 is just a testbed for my hacks so I don't care about recreating the config (and switched to 8.09 with backported packages anyway). Also, the wget2nand script from the ar71xx platform (which I used for reflashing) uses the default yffs formatted partition, so I don't think the mtd -j switch would work anyway (haven't tried it though). But if the rootfs is a jffs, I don't see while config-keeping shouldn't work (ok, so the tarball would need a symlink or something pointing to the rootfs). Cheers, Malte -- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Malte. I like that .tar.gz idea, it would also cover devices like the Fonera where you have two separate images too. However, it's planned to introduce a unified image format for OpenWrt which would solve most of this issues so I haven't started working on it. But imho it would be great to have such a capability for the stable 8.09 series since it can be added as package on top of the normal code. The question is whether config-keeping is doable in such a way, have you already tried this (with mtd's -j option) ? Regards, JoW -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAko4+kwACgkQdputYINPTPP7bACfdmB2t1AVeYc7vqJ6qMZj3Hql 0nAAnirtOHblYeIrYFeQ4S/Ids7tj92w =qQPA -END PGP SIGNATURE- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi, On Wednesday 17 June 2009 03:38:01 Daniel Dickinson wrote: > No one seems to have responded to this, and I'd like to start working > on it next week. If developers could think about this and comment, it > would be appreciated. I'm not a core OpenWrt developer but was thinking about implementing sysupgrade for the RouterBoard 411. That board has two mtd partitions, one for the kernel and one for the rootfs, so the default way of flashing it with sysupgrade doesn't work. I think most of what you wrote isn't needed, especially the logic on how to write the image. Because that stuff is already included in the platform specific part of sysupgrade and is already included in the old firmware. But if it is neded (like for a new platform), it could be included in my idea as well. The ideas I had for implementing it for the RB411 were quite simple: The firmware image can be a standard .tar.gz; nothing fancy, no own format. A tar would be enough but that one doesn't have a recognizable magic number. The tarball contains a folder named firmware with a bunch of files. Each file is an image which has the same name as the corresponding mtd partition. To introduce md5sums, each file could be accompanied by an optional .md5 file, ie. on the RB411 we'd have /firmware/kernel /firmware/kernel.md5 /firmware/rootfs /firmware/rootfs.md5 A broken tgz header will be recognized by these tools already. If we need any logic for flashing, we can also introduce a sysupgrade dir. Anything in there will go to /lib/upgrade before flashing. For even more flexibility, the root could contain an optional (executable) sysupgrade.sh which called instead of the system sysupgrade. Those were the ideas I had. Cheers, Malte -- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi All, You have some very good ideas. I would like to suggest using [requiring] BOTH md5sum and sha1. This is much harder to fake. I have already seen TWO DIFFERENT VERSIONS of the "same" wireless driver in the wild ??!! It seems that those of us who are successful [openwrt] attract a lot of vultures! "Real time" monitoring of the executable portion of "running" code would probably also be wise. In this way the processor can hopefully eventually notice that code has been "patched" or otherwise changed and cause a reboot. Write once devices look better and better. Comment: There are lots of "smart" people who want to join our party without a proper invitation :)). warm regards to all, Wiz (pen name) On Tue, 16 Jun 2009, Daniel Dickinson wrote: [NON-Text Body part not included] ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
