Hey Apache Users, I'm working on a web application that has a web service component, and a background processor component. Both applications will send messages to the same Kafka topic as an object is manipulated.
In some cases, a web service call in the service component will send a message to Kafka saying key K has state S1, then trigger a background operation, and then the background component will send a message to Kafka saying key K has state S2. However, I'm finding that the topic ends up occasionally having a message K/S2 followed by K/S1, rather than the other way around. As both producers in the web service call and the background processor send messages asynchronously with librdkafka, I believe this is a relatively simple race condition where messages just aren't coming in like I'd like them to. In a consuming Kafka Streams application, I'd be creating a KTable of this topic. What approaches can I take to ensure the the KTable will end up with K/S2 as the state for K, rather than the stale-er K/S1? Would KS reorder messages if they had ordered & coordinated timestamps? If so, how much leeway would it have for S2 being delivered before S1? (I believe librdkafka 0.9.1 doesn't support sending create-time in messages, which makes this is a bit more painful.) Any other approaches that are worth exploring? Thanks for any thoughts, Mathieu
