wget can write a disk image only after the whole file is in RAM. This series lets lwIP wget write the download to a block device while it arrives:
=> wget -o mmc 1 -b 400000 http://192.168.1.65/armbian.img The target is a whole device, a hardware partition, a partition by number or name, and an optional byte offset into it. Patch 1 fixes the buffer size check for a reply without Content-Length; patch 4 sets that check to the target size, and without the fix such a reply would be refused. Patch 2 lets a wget client take the data through a callback, patch 3 adds the block device writer, and patch 4 adds the options to the command. Tested: - ROCK Pi 4A: a 1862270976-byte image to the SD card in 3 min 16 s to 3 min 35 s with a 4 MiB buffer; the first 1862270976 bytes of the card have the sha256 of the image, and the card boots. A 2936012800-byte image, past the Content-Length limit below, took 4 min 18 s and checks and boots the same way. - QEMU arm64, virtio-blk: a 404 writes nothing; a file larger than the target is refused before any write with Content-Length, and at the end of the target without it. - sandbox: ut lib lib_test_sink_blk; every patch builds on sandbox and sandbox64_lwip. Notes: - lwIP's http_client.c reads Content-Length through an int, so from 2 GiB on the length is unknown and a file that does not fit fails only at the end of the target. - EFI HTTP, the other user of check_buffer_size, retries with the reported length when it exceeds the buffer (efi_net.c:1715), so a reply without Content-Length still ends in a 4 GiB allocation there; patch 1 does not change that. - checkpatch asks for IS_ENABLED() at "#if defined(CONFIG_WGET_STREAM)" in cmd/lwip/wget.c; that #if picks lines of the help string, as the WGET_CACERT ones next to it do. gzip and zstd input follow in a separate series on top of this one. Signed-off-by: Igor Velkov <[email protected]> --- Igor Velkov (4): net: lwip: wget: accept a reply without Content-Length in the size check net: lwip: wget: hand the payload to an optional sink lib: add a stream sink for block devices cmd: lwip: wget: write the download to a block device cmd/Kconfig | 9 ++ cmd/lwip/wget.c | 107 +++++++++++++++++++++- configs/sandbox64_lwip_defconfig | 1 + configs/sandbox_defconfig | 1 + doc/usage/cmd/wget.rst | 43 ++++++++- include/net-common.h | 6 ++ include/stream_sink.h | 75 ++++++++++++++++ lib/Kconfig | 8 ++ lib/Makefile | 1 + lib/stream_sink.c | 189 +++++++++++++++++++++++++++++++++++++++ net/lwip/wget.c | 89 +++++++++++++++--- test/lib/Makefile | 3 + test/lib/stream_sink.c | 137 ++++++++++++++++++++++++++++ 13 files changed, 655 insertions(+), 14 deletions(-) --- base-commit: 3401e006432abcbf7d73d7210f10df157f990ed3 change-id: 20261101-b4-wget-blk-r4-833f8a3d344e Best regards, -- Igor Velkov <[email protected]>
