P.S, does my scenario require using windows, or can it be achieved using just KTable?
On Wed, Oct 12, 2016 at 8:56 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Heya, > > Say I'm building a live auction site, with different products. Different > users will bid on different products. And each time they do, I want to > update the product's price, so it should always have the latest price in > place. > > Example: Person 1 bids $3 on Product A, and Person 2 bids $5 on the same > product 100 ms later. > > The second bid arrives first and the price is updated to $5. Then the > first bid arrives. I want the price to not be updated in this case, as this > bid is older than the one I've already processed. > > Here's my understanding of how I can achieve this with Kafka Streaming - > is my understanding correct? > > - I have a topic for receiving bids. The topic has N partitions, and I > have N replicas of my application which hooks up w/ Kafka Streaming, up and > running. > > - I assume each replica of my app will listen to a different partition of > the topic. > > - A user makes a bid on product A. > > - This is pushed to the topic with the key bid_a > > - Another user makes a bid. This is also pushed with the same key (bid_a) > > - The 2nd bid arrives first, and gets processed. Then the first (older) > bid arrives. > > - Because I'm using a KTable, the timestamp of the messages is extracted, > and I'm not shown the older bid because I've already processed the later > bid. The older bid is ignored. > > - All bids on product A go to the same topic partition, and hence the same > replica of my app, because they all have the key bid_a. > > - Because of this, the replica already knows which timestamps it has > processed, and is able to ignore the older messages. > > Is the above understandning correct? > > Also, what will happen if bid 2 arrived and got processed, and then the > particular replica crashed, and was restarted. The restarted replica won't > have any memory of which timestamps it has previously processed. > > So if bid 2 got processed, replica crashed and restarted, and then bid 1 > arrived, what would happen in that case? > > Thanks. >
