When I sent the email last time the message formatting was messed. Hopefully this does not have happen this time.
We have very unique problem. We have an application deployed on WebLogic cluster that is spread across 2 datacenter (active-active) DC1 and DC2 (different LAN but same WAN). This producer app generates different user events, which other apps (consumer apps) are interested in. Right now we use JMS as event storage\messaging. JMS would not scale to leverage all of the events (right now we only pump small percentage of events through JMS), so we want to move Kafka. This nicely fits for our needs. Our current deploy and guideline from network team complicates this very simple use case. 1. Our network team mandates that any deployment be in both DC1 and DC2. So if DC1 is unavailable we switch to DC2. From all the reading I have done, having kafka cluster separate across 2 DC is not recommended, also zookeeper cluster over the wan is going to be slow. So thought is have 2 independent Kafka cluster one in each DC. Then user Mirror Maker (MM) to copy the message from topic on DC1 to DC2 and in other direction as well. (this is based on https://kafka.apache.org/081/ops.html) 2. Also many consumer app will prefer to get message in orderly fashion to make sense of the events. (Ordering provide by kafka is sufficient) But producer app is active-active so our user events would be generate on either DC1 or DC2. How we manage reasonable order between events for user across DC? To address this, the consumer group on DC1 will only consume ODD user id events and DC2 will only consume EVEN user id events. So EVEN user id event generated on DC1, has to go through MM process to DC2. So this introduces the MM processing time and network latency, so ordering is not perfect but I think we can live with it. I am new to Kafka and I appreciate any advice from more seasoned Kafka user. Will this work or fail? Is there better way? Any other suggestions? Thanks, Shri ________________________________ This message and its contents (to include attachments) are the property of National Health Systems, Inc. and may contain confidential and proprietary information. This email and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed. You are hereby notified that any unauthorized disclosure, copying, or distribution of this message, or the taking of any unauthorized action based on information contained herein is strictly prohibited. Unauthorized use of information contained herein may subject you to civil and criminal prosecution and penalties. If you are not the intended recipient, you should delete this message immediately and notify the sender immediately by telephone or by replying to this transmission.
