> wget can write a disk image only after the whole file is in RAM. This > series lets lwIP wget write the download to a block device while it > arrives: > > => wget -o mmc 1 -b 400000 http://192.168.1.65/armbian.img > > The target is a whole device, a hardware partition, a partition by > number or name, and an optional byte offset into it. > > Patch 1 fixes the buffer size check for a reply without Content-Length; > patch 4 sets that check to the target size, and without the fix such a > reply would be refused. Patch 2 lets a wget client take the data > through a callback, patch 3 adds the block device writer, and patch 4 > adds the options to the command. > > Tested: > - ROCK Pi 4A: a 1862270976-byte image to the SD card in 3 min 16 s to > 3 min 35 s with a 4 MiB buffer; the first 1862270976 bytes of the > card have the sha256 of the image, and the card boots. A > 2936012800-byte image, past the Content-Length limit below, took > 4 min 18 s and checks and boots the same way. > - QEMU arm64, virtio-blk: a 404 writes nothing; a file larger than the > target is refused before any write with Content-Length, and at the > end of the target without it. > - sandbox: ut lib lib_test_sink_blk; every patch builds on sandbox and > sandbox64_lwip. > > Notes: > - lwIP's http_client.c reads Content-Length through an int, so from > 2 GiB on the length is unknown and a file that does not fit fails > only at the end of the target. > - EFI HTTP, the other user of check_buffer_size, retries with the > reported length when it exceeds the buffer (efi_net.c:1715), so a > reply without Content-Length still ends in a 4 GiB allocation there; > patch 1 does not change that. > - checkpatch asks for IS_ENABLED() at "#if defined(CONFIG_WGET_STREAM)" > in cmd/lwip/wget.c; that #if picks lines of the help string, as the > WGET_CACERT ones next to it do. > > gzip and zstd input follow in a separate series on top of this one. > > Signed-off-by: Igor Velkov <[email protected]> > --- > Igor Velkov (4): > net: lwip: wget: accept a reply without Content-Length in the size check > net: lwip: wget: hand the payload to an optional sink > lib: add a stream sink for block devices > cmd: lwip: wget: write the download to a block device > > cmd/Kconfig | 9 ++ > cmd/lwip/wget.c | 107 +++++++++++++++++++++- > configs/sandbox64_lwip_defconfig | 1 + > configs/sandbox_defconfig | 1 + > doc/usage/cmd/wget.rst | 43 ++++++++- > include/net-common.h | 6 ++ > include/stream_sink.h | 75 ++++++++++++++++ > lib/Kconfig | 8 ++ > lib/Makefile | 1 + > lib/stream_sink.c | 189 > +++++++++++++++++++++++++++++++++++++++ > net/lwip/wget.c | 89 +++++++++++++++--- > test/lib/Makefile | 3 + > test/lib/stream_sink.c | 137 ++++++++++++++++++++++++++++ > 13 files changed, 655 insertions(+), 14 deletions(-) > --- > base-commit: 3401e006432abcbf7d73d7210f10df157f990ed3 > change-id: 20261101-b4-wget-blk-r4-833f8a3d344e > > Best regards,
Applied to u-boot-net branch for-main [1], thanks! [1] https://git.u-boot-project.org/u-boot/custodians/u-boot-net/-/commits/for-main [1/4] net: lwip: wget: accept a reply without Content-Length in the size check commit: 8498fbf8788eff07bc7fc59f9b15cf6c65025b10 [2/4] net: lwip: wget: hand the payload to an optional sink commit: 40611fe0277a7a8f9639aab02a6f1a1e0e897d57 [3/4] lib: add a stream sink for block devices commit: fdf4d9621c8b507f69fe0dd09fc53ebfe860fda5 [4/4] cmd: lwip: wget: write the download to a block device commit: f0870bb385cf923bfab0baddacc952151efa4c39 Best regards, -- Jerome Forissier <[email protected]>
