Paige and I threw this together yesterday morning when we had a free hour - it attempts to twitter new civilian casualties in iraq.
Perhaps folks can suggest ways to improve it - it is kind of brute force - mapping the deaths could also be done - but I'm assuming third party services already do that and the location is published with the twitter albeit using David Troy's notation. - me ---------- Forwarded message ---------- From: paige saez <[EMAIL PROTECTED]> Date: Tue, Sep 9, 2008 at 1:33 PM Subject: war tweets- To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> http://twitter.com/IRAQDEATHS #!/usr/local/bin/ruby # # we want to fetch posts from the iraq death count and show them # and we want to only check every day or so ( an external cron job ) # basically then we take a look at the most recent 50 only... # and we twitter up to 50 of them basically... # we reverse the twitter order so that the most recent ones are first # twittercap = 50 # twitter this many posts max # # 1) fetch the data # # http://www.iraqbodycount.org/database/download/ibc-individuals require 'net/http' url = "http://www.iraqbodycount.org/database/download/ibc-individuals"; data = Net::HTTP.get_response(URI.parse(url)).body # # 2) chew on the data a bit - masticate it good # # our first goal is to read the rather dorky csv file iraq war deaths provides # we have to skip past the header crap... so lets make a copy of the data without that header # we will just loop through all the data and pull out the good stuff and store it into another array # clearly not the "perfectly clean" way to do it but good enough and we don't really care # # http://www.rubytips.org/2008/01/06/csv-processing-in-ruby/ # http://fastercsv.rubyforge.org/classes/FasterCSV.html require 'rubygems' require "fastercsv" @results = FasterCSV.parse(data) # unused approach -> read("ibc-individuals") @deaths = [] inside_stupid_header = true @results.each do |death| if death[0] == "IBC code" inside_stupid_header = false elsif inside_stupid_header == false @deaths << death end end # # 3) put the data in our storage area... keep a copy in our belly # # http://sqlite-ruby.rubyforge.org/sqlite3/faq.html # # our second goal is to store this in a structured way so we can process it # i will bother to keep it in a database although i could just hold it memory # we are going to want to avoid storing duplicates - because we will call this multiple times # # this gets slightly more involved # # 3a) let us make a table to store the data - if it already exists this code will crash but thats ok # we wrap the whole thing in a begin / rescue and just ignore the crash if it happens # # The format seems to be here in the ibc file : "IBC code","Name or Identifying Details","Age","Sex","Marital Status","Parental Status","Earliest Date","Latest Date","Location" # # 3b) um, store everything... =begin require 'rubygems' require 'sqlite' # this was a manual database approach - tedious! db = SQLite::Database.new( "endiraqwar.db" ) begin result = db.execute("CREATE TABLE deaths(id INTEGER, code VARCHAR(80), name VARCHAR(255), age INTEGER, sex VARCHAR(32), marital VARCHAR(64), parental VARCHAR(64), earliest DATE, latest DATE, location VARCHAR(255))"); rescue end @deaths.each do |death| result = db.execute("INSERT INTO ") end =end # # 3) ok scratch the above, lets use this: # # http://datamapper.org/why.html # # lets manage the data in a way that takes advantage of object oriented design # we will grab datamapper which lets us pretend our data is a ruby data object # (instead of treating it like radioactive waste and holding it at arms length) # that means we have to define what our data "is" for ruby... # also we will hop over to postgres as our back end datastore - away from sqlite ... just because # require 'rubygems' require 'dm-core' DataMapper.setup(:default, { :adapter => 'postgres', :database => "endiraqwar", :username => 'endiraqwar', :password => '', :host => 'localhost' }) class Death include DataMapper::Resource property :id, Integer, :serial => true property :code, String property :name, Text property :age, Text property :sex, Text property :marital, Text property :parental, String property :earliest, DateTime property :latest, DateTime property :location, Text property :created_at, DateTime property :posted, DateTime, :default => nil end # we do not want to do this now # because it would erase our database # DataMapper.auto_migrate! # go ahead and store all the deaths @deaths.each do |death| if Death.first(:code => death[0] ) # puts "We already found this death #{death[1]} #{death[0]} so not saving" next end # take a second to convert the date phrase into a machine date death[6] = DateTime.parse(death[6]) death[7] = DateTime.parse(death[7]) # go ahead and make a blobby thing to hold all of this record = Death.new( :code => death[0], :name => death[1], :age => death[2], :sex => death[3], :marital => death[4], :parental => death[5], :earliest => death[6], :latest => death[7], :location => death[8] ) puts "recording the passing of #{record.name} at #{record.earliest} and #{record.code}" record.save end # # 4) ok cool, now we have the data - lets twitter! # # http://twitter.rubyforge.org/ # http://github.com/jnunemaker/twitter/tree/master/examples/posting.rb # require 'twitter' twitter = Twitter::Base.new("endiraqwar","") @deaths = Death.all(:order => [:earliest.desc], :limit => twittercap) @copyofdeaths = [] @deaths.each do |death| @copyofdeaths << death end @copyofdeaths.reverse.each do |death| # publish deaths that are new next if death.posted != nil result = twitter.post("#{death.name}, #{death.age}, #{death.sex}, #{death.marital}, #{death.parental} killed on #{death.earliest.strftime("%d %b %Y")} at L:#{death.location}") # remember that we already published this death death.posted = DateTime.now death.save puts "posted the death of #{death.name} #{death.code}" end -- Paige Saez 971.227.4384 Art.Media.Love. paigesaez.org makerlab.com red76.com -- anselm 415 215 4856 http://hook.org http://makerlab.com http://meedan.net
_______________________________________________ Geowanking mailing list [email protected] http://lists.burri.to/mailman/listinfo/geowanking
