Hi, gzip/zlib compression is very bad for performance, so it can be interesting for closed indices, but for live data I would not recommend it. Also, you must know that:
Compression using lz4 is already enabled into indices, ES/Lucene/Java usually read&write 4k blocks, -> hence, compression is achieved on 4k blocks. If your filesystem uses 4k blocks and you add FS compression, you will probably have a very small gain, if any. I've tried on ZFS: Filesystem Size Used Avail Capacity Mounted on zdata/ES-lz4 1.1T 1.9G 1.1T 0% /zdata/ES-lz4 zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES If you are using a larger block size, like 128k, a compressed filesystem does show some benefit: Filesystem Size Used Avail Capacity Mounted on zdata/ES-lz4 1.1T 1.1G 1.1T 0% /zdata/ES-lz4 -> compressratio 1.73x zdata/ES-gzip 1.1T 901M 1.1T 0% /zdata/ES-gzip -> compressratio 2.27x zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES But a file system block larger than 4k is very suboptimal for IO (ES read or write one 4k block -> your FS must read or write a 128k block). On 21 juil. 2014, at 07:58, horst knete <baduncl...@hotmail.de> wrote: > Hey guys, > > we have mounted an btrfs file system with the compression method "zlib" for > testing purposes on our elasticsearchserver and copied one of the indices > on the btrfs volume, unfortunately it had no success and still got the size > of 50gb :/ > > I will further try it with other compression methods and will report here > > Am Samstag, 19. Juli 2014 07:21:20 UTC+2 schrieb Otis Gospodnetic: >> >> Hi Horst, >> >> I wouldn't bother with this for the reasons Joerg mentioned, but should >> you try it anyway, I'd love to hear your findings/observations. >> >> Otis >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> >> On Wednesday, July 16, 2014 6:56:36 AM UTC-4, horst knete wrote: >>> >>> Hey Guys, >>> >>> to save a lot of hard disk space, we are going to use an compression file >>> system, which allows us transparent compression for the es-indices. (It >>> seems like es-indices are very good compressable, got up to 65% >>> compression-rate in some tests). >>> >>> Currently the indices are laying at a ext4-Linux Filesystem which >>> unfortunately dont have the transparent compression ability. >>> >>> Anyone of you got experience with compression file systems like BTRFS or >>> ZFS/OpenZFS and can tell us if this led to big performance losses? >>> >>> Thanks for responding -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3DD72EC1-E3EC-493D-94DD-33E63151A579%40patpro.net. For more options, visit https://groups.google.com/d/optout.
