What sort of data are you indexing? When you said performance impact was minimal, how minimal and at what points are you seeing it?
Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 4 August 2014 16:43, horst knete <baduncl...@hotmail.de> wrote: > Hi again, > > a quick report regarding compression: > > we are using a 3-TB btrfs-volume with 32k block size now which reduced the > amount of data from 3,2 TB to 1,1TB without any segnificant performance > losses ( we are using a 8 CPU, 20 GB Memory machine with an iSCSI.Link to > the volume ). > > So for us i can only suggest using the btrfs-volume for long term storage. > > Am Montag, 21. Juli 2014 08:48:12 UTC+2 schrieb Patrick Proniewski: >> >> Hi, >> >> gzip/zlib compression is very bad for performance, so it can be >> interesting for closed indices, but for live data I would not recommend it. >> Also, you must know that: >> >> Compression using lz4 is already enabled into indices, >> ES/Lucene/Java usually read&write 4k blocks, >> >> -> hence, compression is achieved on 4k blocks. If your filesystem uses >> 4k blocks and you add FS compression, you will probably have a very small >> gain, if any. I've tried on ZFS: >> >> Filesystem Size Used Avail Capacity Mounted on >> zdata/ES-lz4 1.1T 1.9G 1.1T 0% /zdata/ES-lz4 >> zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES >> >> If you are using a larger block size, like 128k, a compressed filesystem >> does show some benefit: >> >> Filesystem Size Used Avail Capacity Mounted on >> zdata/ES-lz4 1.1T 1.1G 1.1T 0% >> /zdata/ES-lz4 -> compressratio 1.73x >> zdata/ES-gzip 1.1T 901M 1.1T 0% >> /zdata/ES-gzip -> compressratio 2.27x >> zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES >> >> But a file system block larger than 4k is very suboptimal for IO (ES read >> or write one 4k block -> your FS must read or write a 128k block). >> >> On 21 juil. 2014, at 07:58, horst knete <badun...@hotmail.de> wrote: >> >> > Hey guys, >> > >> > we have mounted an btrfs file system with the compression method "zlib" >> for >> > testing purposes on our elasticsearchserver and copied one of the >> indices >> > on the btrfs volume, unfortunately it had no success and still got the >> size >> > of 50gb :/ >> > >> > I will further try it with other compression methods and will report >> here >> > >> > Am Samstag, 19. Juli 2014 07:21:20 UTC+2 schrieb Otis Gospodnetic: >> >> >> >> Hi Horst, >> >> >> >> I wouldn't bother with this for the reasons Joerg mentioned, but >> should >> >> you try it anyway, I'd love to hear your findings/observations. >> >> >> >> Otis >> >> -- >> >> Performance Monitoring * Log Analytics * Search Analytics >> >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> >> >> >> >> >> On Wednesday, July 16, 2014 6:56:36 AM UTC-4, horst knete wrote: >> >>> >> >>> Hey Guys, >> >>> >> >>> to save a lot of hard disk space, we are going to use an compression >> file >> >>> system, which allows us transparent compression for the es-indices. >> (It >> >>> seems like es-indices are very good compressable, got up to 65% >> >>> compression-rate in some tests). >> >>> >> >>> Currently the indices are laying at a ext4-Linux Filesystem which >> >>> unfortunately dont have the transparent compression ability. >> >>> >> >>> Anyone of you got experience with compression file systems like BTRFS >> or >> >>> ZFS/OpenZFS and can tell us if this led to big performance losses? >> >>> >> >>> Thanks for responding >> > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/1f9bf509-b185-4c66-99c5-d8f69e95bea8%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/1f9bf509-b185-4c66-99c5-d8f69e95bea8%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ayiQiTBeiNkn5NKRqm2Vyu5Bk2md8Gw%2BC7nSVJXF9%2BGw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
