[ https://issues.apache.org/jira/browse/KAFKA-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487915#comment-13487915 ]
Neha Narkhede commented on KAFKA-595: ------------------------------------- >> I would recommend we instead add a log.compression.codec property (plus >> override map) that controls the compression on the broker. Yes, when I said move the compression to server side, that is what I meant. Also, I filed this JIRA to keep track of this optimization I came across while performance testing, don't think we should push this in 0.8 > Producer side compression is unnecessary > ---------------------------------------- > > Key: KAFKA-595 > URL: https://issues.apache.org/jira/browse/KAFKA-595 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.8 > Reporter: Neha Narkhede > Labels: feature, features > > Compression can be used to store something in less space (less IO) and/or > transfer it less expensively (better use of network bandwidth). Often the two > go hand in hand, such as when compressed data is written to a disk: the disk > I/O takes less time, since less bits are being transferred, and the storage > occupied on the disk after the transfer is less. Unfortunately, the time to > compress the data can exceed the savings gained from transferring less data, > resulting in overall degradation. > After KAFKA-506, the network usage gains we used to get by compressing data > at the producers is exceeded by the cost of decompressing and re-compressing > data at the server side. Compression to save on network costs must be done > either to reduce the contention in a wide-area network due to multiple point > to point connections OR to efficiently transfer data over low-bandwidth > networks (cross DC). In the case of producer-server connections, neither is > typically true, which means we might not benefit from producer side > compression at all in most production deployments of Kafka. On the contrary, > it might actually hurt performance since most production deployments turn on > compression for all topics. > The main benefit of compressing data in Kafka is to efficiently transfer data > cross DC for setting up mirrored Kafka clusters. The performance benefit is > also true for real time consumers, especially when there are multiple groups > of consumers consuming the same topic. If data is compressed on the server > side instead, which we do anyways, we can get the I/O savings as well as > efficient network transfer on the server-consumer links. > I don't have numbers to quantify the performance impact of re-compression > now, since there are other changes that need to be done to test this > correctly. > Thoughts ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira