On Wed, Dec 17, 2014 at 12:12 AM, Merlin Moncure <mmonc...@gmail.com> wrote:
> Compressing *one* file with lz4 and a quick/n/dirty plgz i hacked out > of the source (borrowing heavily from > https://github.com/maropu/pglz_bench/blob/master/pglz_bench.cpp), I > tested the results: > > lz4 real time: 0m0.032s > pglz real time: 0m0.281s > > mmoncure@mernix2 ~/src/lz4/lz4-r125 $ ls -lh test.* > -rw-r--r-- 1 mmoncure mmoncure 2.7M Dec 16 09:04 test.lz4 > -rw-r--r-- 1 mmoncure mmoncure 2.5M Dec 16 09:01 test.pglz > A better test would examine all manner of different xlog files in a > fashion closer to how your patch would need to compress them but the > numbers here tell a fairly compelling story: similar compression > results for around 9x the cpu usage. > Yes that could be better... Thanks for some real numbers that's really informative. Be advised that compression alg > selection is one of those types of discussions that tends to spin off > into outer space; that's not something you have to solve today. Just > try and make things so that they can be switched out if things > change.... > One way to get around that would be a set of hooks to allow people to set up the compression algorithm they want: - One for buffer compression - One for buffer decompression - Perhaps one to set up the size of the buffer used for compression/decompression scratch buffer. In the case of pglz, this needs for example to be PGLZ_MAX_OUTPUT(buffer_size). The part actually tricky is that as xlogreader.c is used by pg_xlogdump, we cannot directly use a hook in it, but we may as well be able to live with a fixed maximum size like BLCKSZ * 2 by default, this would let largely enough room for all kinds of compression algos IMO... Regards, -- Michael
