On Tue, Jul 09, 2019 at 08:04:23PM +0300, Leonid Bobrov wrote: > Stuart and Espie, have you ever heard of compression?
once I ran out of space on the 80MB drive my BBS was running on so I went through the whole download area, compressing everything with arj/lha/zip using various flags (sometimes "lower level" compression actually gave smaller files) and keeping the smallest file. so yeah I've heard of compression. it took *ages*. and then it got blown up by lightning anyway. On 2019/07/09 20:40, Marc Espie wrote: > Most specifically: > - we do use gzip because other compression systems won't work with little > memory/don't have the right licence. > - gzip allows you to stick COMMENTS in the header, which is where the > signature lives. I DID A WHOLE FUCKING PRESENTATION ABOUT THAT DESIGN > CHOICE. > - pkg_create uses an LRU cache to make updates faster. And we use some > specificities of gzip to make packages amenable to using rsync actually. I am quite interested in zstd (RFC8478) though.. it does seem to have a lot of the characteristics we need: + bsd license + compressed files can be concatenated like gzip + "skippable frames", the signature could probably live here + usually better compression than gzip, even with zstd -1 + so much faster than nearly everything else it isn't funny - it does use a little more ram than gzip but not hugely, and with library_aslr and kernel relinking we've pushed out the minimum usable memory already. max rss running *cat on a test package tar (varies a bit depending on compression level): gz 1.5M zstd 3M lz4 10M On 2019-07-09, Nathan Hartman <hartman.nat...@gmail.com> wrote: > You might find this interesting: > https://github.com/silentbicycle/heatshrink interesting for a very constrained system, but we are ok with a couple of MB rather than a couple of KB ram use, and it wouldn't be worth moving to something with lower compression ratios than we already have.
