You could use SQS for an outbound heartbeat poll from each server. If the primary stops responding the secondary takes over. SQS is transactional. In this circumstance you don't have two streams running concurrently reducing resource usage. Depending on how frequently you poll you might lose a bit of data which might rule this option out for you.
Regards, Patrick Sent from my iPhone On Jan 10, 2013, at 7:30 AM, blindman2k <[email protected]> wrote: > Hello beachers, > > I am reaching out to the brains trust for a solution to an age-old problem. I > have two "identical" AWS EC2 servers streaming tweets from Twitter and > putting the results of a filter into an end process via AWS SQS. The data > should effectively be identical although the timing is not guaranteed to be. > What's the most effective and efficient way to get only one copy of the data > to the end process? I am leaning towards writing each event (or a hash) into > a unique-keyed database table to detect and remove duplicates but I am > looking for alternative ideas. > > Anyone? > > -- > You received this message because you are subscribed to the Silicon Beach > Australia mailing list. Vist http://siliconbeachaustralia.org for more > > Forum rules > 1) No lurkers! It is expected that you introduce yourself. > 2) No jobs postings. You can use http://siliconbeachaustralia.org/jobs > > > To post to this group, send email to > [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en -- You received this message because you are subscribed to the Silicon Beach Australia mailing list. Vist http://siliconbeachaustralia.org for more Forum rules 1) No lurkers! It is expected that you introduce yourself. 2) No jobs postings. You can use http://siliconbeachaustralia.org/jobs To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en
