[twitter-dev] how are people collecting spritzer/gardenhose?

[Brendan O'Connor]

spritzer is great!  well done folks.
I'm wondering how other people are collecting the data.  I'm saving the
json-per-line raw output to a flatfile, just using a restarting curl, then
processing later.

Something as simple as this seems to work for me:

while true; do
  date; echo "starting curl"
  curl -s -u user:pass http://stream.twitter.com/spritzer.json >>
tweets.$(date --iso)
  sleep 1
done |& tee curl.log

... and also, to force file rotation once in a while:

while true; do
  date; echo "forcing curl restart"
  killall curl
  sleep $((60*60*5))
done |& tee kill.log


anyone else?

-Brendan