Wehave very unique problem. Wehave a application deployed on weblogic cluster that is spread across 2 datacenter(active-active) DC1 and DC2 (different LAN but same WAN). This producer app generatesdifferent user events, which other apps (consumer apps) are interested in.Right now we use JMS as event storage\messaing. JMS would not scale to leverage all ofthe events (right now we only pump small percentage of events through JMS), so wewant to move Kafka. This nicely fits for our needs.
Ourcurrent deploy and guideline from network team complicates this very simple usecase. 1. Our network teammandates that any deployment be in both DC1 and DC2. So if DC1 is unavailable weswitch to DC2. From all the reading I have done, having kafka cluster separateacross 2 DC is not recommended, also zookeeper cluster over the wan is going tobe slow. So thought is have 2 independent kafkacluster one in each DC. Then user Mirror Maker (MM) to copy the message from topic onDC1 to DC2 and in other direction as well. (this is based on https://kafka.apache.org/081/ops.html) 2. Also many consumerapp will prefer to get message in orderly fashion to make sense of the events.(ordering provide by kafka is sufficient) But producer app is active active so our userevents would be generate on either DC1 or DC2. How we manage reasonable orderbetween events for user across DC? To address this, the consumer group on DC1will only consume ODD user id events and DC2 will only consume EVEN user id events. So EVEN user id event generated on DC1, hasto go through MM process to DC2. So this introduces the MM processing time andnetwork latency, so ordering is not perfect but I think we can live with it. Iam new to kafka and I appreciate any advice from more seasoned kafka user. Will this work or fail? Is there better way? Any other suggestions? Thanks, Shri
