Joris Dobbelsteen wrote: > Also I still have to see an compression algorithm that can sustain over > (or even anything close to, for that matter) 100MB/s on todays COTS > hardware. As TOAST provides compression, maybe that data can be > transmitted in compressed manner (without recompression).
I did a few quick tests on compression speed, as I was curious about just what sort of performance was available. I was under the impression that modern hardware could easily top 100 Mbit/s with common compression algorithms, and wanted to test that. Based on the results I'd have to agree with the quoted claim. I was apparently thinking of symmetric encryption throughput rather than compression throughput. I get 19 Mbit/s from gzip (deflate) on my 2.4GHz Core 2 Duo laptop. With lzop (LZO) the machine achieves 45 Mbit/s. In both cases only a single core is used. With 7zip (LZMA) it only manages 3.1 Mb/s using BOTH cores together. All tests were done on a 278MB block of data that was precached in RAM. Output was to /dev/null except for the LZMA case (due to utility limitations) in which case output was written to a tmpfs. Perhaps a multi-core and/or SIMD-ized implementation of LZO (if such a thing is possible or practical) might manage 100 Mbit/s, or you might pull it off on an absolutely top of the range desktop (or server) CPU like the 3.3 GHz Core 2 Duo. Maybe, but probably not without considerable overclocking, which eliminates the "COTS" aspect rather soundly. Given that very few people have dedicated gzip (or other algorithm) acceleration cards in their systems, it looks like it should be faster to do transfers uncompressed over a network of any respectable speed. Not entirely surprising, really, or it'd be used a lot more in common file server protocols. Wire protocol compression support in PostgreSQL would probably still be extremely useful for Internet or WAN based clients, though, and there are probably more than a few of those around. I know it'd benefit me massively, as I have users using PostgreSQL over 3G cellular radio (UMTS/HSDPA) where real-world speeds are around 0.1 - 1.5 Mbit/s, data transfer limits are low and data transfer charges are high. Compression would clearly need to be a negotiated connection option, though. Interestingly, the Via thin clients at work, which have AES 256 (among other things) implemented in hardware can encrypt to AES 256 at over 300 MB/s. Yes, megabytes, not megabits. Given that the laptop used in the above testing only gets 95 MB/s, it makes you wonder about whether it'd be worthwhile for CPU designers to offer a common compression algorithm like LZO, deflate, or LZMA in hardware for server CPUs. -- Craig Ringer -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
