So, we have lots of apps producing messages to our kafka 0.7.2 instances (and multiple consumers of the data).
We are not going to be able to follow the suggested migration path, where we first migrate all data, then move all producers to use 0.8, etc. Instead, many apps are on their own release cycle, and we need to allow them to upgrade their kafka libraries as part of their regular release schedule. Is there a procedure I'm not seeing, or am I right in thinking I'll need to maintain duplicate kafka clusters (and consumers) for a time. Or can we have a real-time data migration consumer always running continuously against the 0.7.2 kafka store, and have all the data ultimately end up in 0.8. Eventually, the data going to 0.7.2 will dwindle to nothing, but it could take a while. So, I'm thinking I'll just need to maintain dual sets of kafka servers for a while. Since this won't result in any increase in load/disk space, etc., I was thinking of allowing instances to remain multi-tennant with each other (e.g. kafka 0.7.2 and kafka 0.8 on the same box, using separate ports, separate log storage directories (but shared disks)). Is this ok, or a terrible idea? I expect the transition to take several weeks. Thanks, Jason
