I am using python to implement a process which listens to the stream
and places all incoming data onto a message queue service. A few
other worker processes in the background work off the queue and store
the data. The message queue is not fault tollerant at this time,
however with a simple switch to an enterprise based MQ service that
could be achieved.
You are essentially doing the same thing via some bash scripts and
flatfiles. How are you parsing and indexing the data once its
collected?
On May 25, 5:02 pm, Brendan O'Connor breno...@gmail.com wrote:
spritzer is great! well done folks.
I'm wondering how other people are collecting the data. I'm saving the
json-per-line raw output to a flatfile, just using a restarting curl, then
processing later.
Something as simple as this seems to work for me:
while true; do
date; echo starting curl
curl -s -u user:passhttp://stream.twitter.com/spritzer.json
tweets.$(date --iso)
sleep 1
done | tee curl.log
... and also, to force file rotation once in a while:
while true; do
date; echo forcing curl restart
killall curl
sleep $((60*60*5))
done | tee kill.log
anyone else?
-Brendan
