Marc A. Lehmann on 2005-02-12 19:23:55 +0100: > Hmm, nothing in the man page claims it's copying that much into > memory, or that it needs that much memory. It does refer to "900MB > history buffer",
For the history buffer to be effective, it must be in memory and can't be piecemeal loaded and discarded. For a similar reason, rzip can't be used in a pipe. The author is not under the impression it can be easily coded around. > so maybe that means it needs that much memory (in fatc, strace does > not show that it tries to allocate that much memory, all it dos is > mmap the file, but the same could be done by read()ing it). As mentioned, the history buffer needs to be in memory; I don't see the advantage in read()ing it instead of mmap()ing it, since the end result is that up to 900MB of the file is in memory at the same time. > If that is indeed the case, that explains the problem. It might help > if that gets documented more clearly in the manpage. I somehow doubt > that it needs 900MB of ram, though. It doesn't *really* need 900MB of RAM + swap, as I read the code; take a look at rzip.c:581 if you're so inclined. The minimum history buffer size is 100MB, and each additional level of compression adds another 100MB of memory. In your original example, you used `rzip -9`. I am under the impression that this is how bzip2 works. The man page could be a little clearer. If you would like to submit a diff, that would be great; otherwise, I'll take care of it as time permits.
pgpKTLVRWclhe.pgp
Description: PGP signature
