i run tests with this command: bin/kafka-producer-perf-test.sh --topic topicname --num-records 500000000 --throughput -1 --producer.config config/producer.properties --record-size 64
śr., 12 wrz 2018 o 13:38 Liam Clarke <liam.cla...@adscale.co.nz> napisał(a): > 500 million * 64B is 32GB. Are you sure you actually sent 500 million > messages? (I assumed that mln = million) > > On Wed, 12 Sep. 2018, 9:54 pm darekAsz, <darekka...@gmail.com> wrote: > > > sorry, I wrote bad results :/ > > here are correctly > > > > Directory size after sending uncompressed data: 1.5 GB > > > > Directory size after sending data compressed with gzip: 1.2 GB > > > > Directory size after sending data compressed with snappy: 366 MB > > > > Directory size after sending data compressed with lz4: 2.4 GB > > > > > > So gzip and snappy compress data, but lz4 increases data size > > > > śr., 12 wrz 2018 o 11:33 Chanchal Chatterji < > > chanchal.chatte...@infosys.com> > > napisał(a): > > > > > It's not just lz4 , except in case if gzip everything else increases > the > > > directory size. > > > > > > -----Original Message----- > > > From: darekAsz <darekka...@gmail.com> > > > Sent: Wednesday, September 12, 2018 2:43 PM > > > To: users@kafka.apache.org > > > Subject: Kafka compression - results > > > > > > Hi > > > I made some tests of compression in kafka. First I want to check speed > of > > > producer with compression. There are my results: > > > with no compression: 112.58 MB/s > > > with gzip compression: 63.24 MB/s > > > with snappy compression: 132.43 MB/s > > > with lz4 compression: 136.66 MB/s > > > > > > Then I want to check how looks size of data on broker. I sent 500 mln > 64B > > > records. There are my results: > > > directory size before sending data: 4KB > > > directory size after sending data compressed with gzip: 1.5 GB > directory > > > size after sending data compressed with snappy: 366 MB directory size > > after > > > sending data compressed with lz4: 2.4 GB > > > > > > And there is my question: why lz4 compression make data size bigger > than > > > data without compression? I think that i made all tests correctly. > > > > > > Best regards > > > > > >
