Interesting. Elasticache should enable you to see if a piece of data has been processed as it acts as memcache that is common across all your EC2 instances, it is very easy to configure and use. If you take a hash of the data and use that as a unique key on insert into your DB, you should avoid duplicates no problem.
I am surprised you say you can't deal with delays of more than a few milliseconds but you're using SQS though? It's not designed to be quick. Given you only have two servers and the events are rare and tightly timed, you could also look for a consensus between the EC2 instances before sending to SQS or direct to your DB. Not sure what you code in, but Go does this type of stuff really elegantly, it also handles queuing internally with buffered channels and crunches data at almost C speed. Good luck! Hamish On Friday, 11 January 2013 11:43:01 UTC+11, blindman2k wrote: > > Thanks for the great responses ... some more detailed parameters have to > be drawn out: > > > 1. I can't tolerate delays of more than a few milliseconds. > 2. I can't lose events. > 3. I can't tolerate duplicates (although there is some level of > duplicate protection downstream, I can't rely on it). > 4. The events are not very frequent, say one every 5-10 minutes. > 5. memcache was an early thought but it introduces another failure > point and another configuration item. Given the previous point (the events > are not very frequent), a transactional database should be a sufficient > replacement. > > -- You received this message because you are subscribed to the Silicon Beach Australia mailing list. Vist http://siliconbeachaustralia.org for more Forum rules 1) No lurkers! It is expected that you introduce yourself. 2) No jobs postings. You can use http://siliconbeachaustralia.org/jobs To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en
