> On Dec 2, 2016, at 4:57 PM, Michael Pearce <michael.pea...@ig.com> wrote: > > Hi Jun. > > RE Mirroring, > > [...] > > Lastly around mirroring we have a partionKey field, as the key used for > portioning logic != compaction key all the time but we want to preserve it > for when we mirror so that if source cluster partition count != destination > cluster partition count we can honour the same partitioning logic. > >
Michael, Sorry to be off topic about the KIP, but this is genius. I'm totally stealing this idea. We currently require partition counts to be the same on source and destination cluster and do 1-to-1 mapping of partitions (i.e. partition 1 goes to partition 1, partition 2 goes to partition 2, etc) because otherwise mirrormaker would partition based off of the compaction key. Explicitly including the partition key would let us work around that. Does that mean you have a custom partitioner in mirrormaker that uses the partitionKey field? Is it implemented inside a MirrorMakerMessageHandler? Thanks! -James
