Hi, I'm building a topology where I am connecting twitter with another application (ex: appA)
appA also consists of the following graph model (similar to facebook/twitter) where a user can have followers and follow other users. Ex: UserA follows UserB, UserC, UserD. And UserB/C/D can have any no.of followers. This information is currently stored in an Oracle table. I am retrieving the corresponding twitter ids for users B,C and D and retrieving the latest (n) tweets posted by them. 1) I have a Kafka Spout where I am streaming the tweets for a specific set of userIds. 2) I do a join of the Kafka Spout with the records in oracle table in another Bolt, so that each tweet would be joined with all the users who follow the user who posted the particular tweet. 3) After doing a join, I 'll be using a RollingCountBolt to capture the latest n tweets posted by all the followers for a given user. My question is what is the best way to store the results of RollingCountBolt by avoiding duplication. I can use a Redis instance to capture the information. But say if a userA is followed by 100 users, a tweet posted by userA will be duplicated 100 times. To avoid duplication I can just store tweetIds in the outputField of RollingCountBolt and tweets can be stored in a separate table. But since tweets are streaming, each record must be associated with an expiration period while being stored (similar to cache). How are such scenarios dealt with usually in a streaming application?Suggestions would be helpful. Thanks Kanagha
