Hi again, a quick report regarding compression:
we are using a 3-TB btrfs-volume with 32k block size now which reduced the amount of data from 3,2 TB to 1,1TB without any segnificant performance losses ( we are using a 8 CPU, 20 GB Memory machine with an iSCSI.Link to the volume ). So for us i can only suggest using the btrfs-volume for long term storage. Am Montag, 21. Juli 2014 08:48:12 UTC+2 schrieb Patrick Proniewski: > > Hi, > > gzip/zlib compression is very bad for performance, so it can be > interesting for closed indices, but for live data I would not recommend it. > Also, you must know that: > > Compression using lz4 is already enabled into indices, > ES/Lucene/Java usually read&write 4k blocks, > > -> hence, compression is achieved on 4k blocks. If your filesystem uses 4k > blocks and you add FS compression, you will probably have a very small > gain, if any. I've tried on ZFS: > > Filesystem Size Used Avail Capacity Mounted on > zdata/ES-lz4 1.1T 1.9G 1.1T 0% /zdata/ES-lz4 > zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES > > If you are using a larger block size, like 128k, a compressed filesystem > does show some benefit: > > Filesystem Size Used Avail Capacity Mounted on > zdata/ES-lz4 1.1T 1.1G 1.1T 0% > /zdata/ES-lz4 -> compressratio 1.73x > zdata/ES-gzip 1.1T 901M 1.1T 0% > /zdata/ES-gzip -> compressratio 2.27x > zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES > > But a file system block larger than 4k is very suboptimal for IO (ES read > or write one 4k block -> your FS must read or write a 128k block). > > On 21 juil. 2014, at 07:58, horst knete <badun...@hotmail.de <javascript:>> > wrote: > > > Hey guys, > > > > we have mounted an btrfs file system with the compression method "zlib" > for > > testing purposes on our elasticsearchserver and copied one of the > indices > > on the btrfs volume, unfortunately it had no success and still got the > size > > of 50gb :/ > > > > I will further try it with other compression methods and will report > here > > > > Am Samstag, 19. Juli 2014 07:21:20 UTC+2 schrieb Otis Gospodnetic: > >> > >> Hi Horst, > >> > >> I wouldn't bother with this for the reasons Joerg mentioned, but should > >> you try it anyway, I'd love to hear your findings/observations. > >> > >> Otis > >> -- > >> Performance Monitoring * Log Analytics * Search Analytics > >> Solr & Elasticsearch Support * http://sematext.com/ > >> > >> > >> > >> On Wednesday, July 16, 2014 6:56:36 AM UTC-4, horst knete wrote: > >>> > >>> Hey Guys, > >>> > >>> to save a lot of hard disk space, we are going to use an compression > file > >>> system, which allows us transparent compression for the es-indices. > (It > >>> seems like es-indices are very good compressable, got up to 65% > >>> compression-rate in some tests). > >>> > >>> Currently the indices are laying at a ext4-Linux Filesystem which > >>> unfortunately dont have the transparent compression ability. > >>> > >>> Anyone of you got experience with compression file systems like BTRFS > or > >>> ZFS/OpenZFS and can tell us if this led to big performance losses? > >>> > >>> Thanks for responding > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1f9bf509-b185-4c66-99c5-d8f69e95bea8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
