Hi, Chris, Yes, the changes related to replication are significant. It's actually a bit hard to keep what we have now as the degenerated case with R=1. Currently, the partions are numbered locally within each broker. This is problematic with replication since a partition can have multiple replicas that physically exist on multiple brokers. So, in our replication design, partitions will be numbered globally within the cluster. This implies potential on disk layout changes (see kafka-47) and wire protocol changes.
You brought up a good question on migration path from 0.7 to 0.8. Since replication changes are significant, it's going to be hard to make those changes while keeping backward compatibility. My current feeling is that we will make 0.8 a non-backward compatible release. To migrate to 0.8, one possibility is to first create a new cluster of Kafka brokers using 0.8 (can potentially overlay on existing hardware). Then upgrade all producers and point them to the new brokers. Drain all old consumers. Upgrade all consumers and point them to the new brokers. What do people feel about this? Thanks, Jun On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs <chris.burrou...@gmail.com>wrote: > On 11/11/2011 06:01 PM, Jay Kreps wrote: > > 2. Our current focus for linkedin folks was going to be on replication > > which will be a really disruptive set of features that need to be > released > > together. > > I think there are some differing assumptions on the impact of > replication. My assumption has been that replication would be a new > feature that was 'optional' in that R=1 would not be significantly from > what we are doing now. Thus replication work could continue in trunk -- > and even be present in releases -- without being disruptive, the lack > change would just be to allow and document values of R > 1. > > However, both Jay and Jun's description of the work make it sound like > more of a backwards incompatible no-rolling-restart every > broker/consumer/producer needs to be upgraded at the same time sort of > change. >
