You might want to use the new replication quotas mechanism (i.e. network throttling) to make sure that replication traffic doesn't negatively impact your production traffic.
See for details: https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas This feature was added in 0.10.1 -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Mar 14, 2017 at 10:09 AM, Joe San <codeintheo...@gmail.com> wrote: > Dear Kafka Users, > > What are the arguments against setting the retention plociy on a Kafka > topic to infinite? I was in an interesting discussion with one of my > colleagues where he was suggesting to set the retention policy for a topic > to be indefinite. > > So how does this play up when adding new broker partitions? Say, I have > accumulated in my topic some gigabytes of data and now I realize that I > have to scale up by adding another partition. Now is this going to pose me > a problem? The partition rebalance has to happen and I'm not sure what the > implications are with rebalancing a partition that has gigabytes of data. > > Any thoughts on this? > > Thanks and Regards, > Jothi >
