Hi BG,
> This seems to be an ideal use-case for Clojure Agents
> http://clojure.org/agents
This code is still a little too general for me. I guess if I stare at it long
enough, I will eventually get it ;)
> Try it out and share the code :)
Will do, but I hope that someone has something more concrete. Here is what I
do in Ruby (shortened and with all error handling stripped out).
require 'rubygems'
require 'mechanize'
require 'faster_csv'
urls = '...urls in csv format...'
fcsv_options = {:headers => true, :header_converters => :symbol}
FasterCSV.open('results.csv', 'w') do |csv|
csv << %w[url acrank refdomain anchortext title body]
FasterCSV.parse(urls, fcsv_options) do |line|
page = WWW::Mechanize.new { |ag|
ag.read_timeout = 10
}.get(line[:url])
# push result to csv
end
end
a typical spider. I would love to read some source code of something similiar
done in clojure. If it doesn't exist yet, I will eventually create it. Right
now I would even know which libs to use or even if there are any. I guess even
the parsing of the csv could be multithreaded. Feelin kinda frustrated right
now, because I know how to do it in a couple other languages, but just got
started with Clojure. I have done this in Ruby even multithreaded, which
quickly became *very* complicated because of it's mutable variables.
-sasa
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
