I can't really tell from that stacktrace if the problem is this, but
you may be hitting the maximum number of open files (and sockets).
Are you on OSX? If so, you can expand that limit with sysctl. Try:
sudo sysctl -w kern.maxfiles=36000
Sets to a bit over twice as much the default. Also, check
kern.maxfilesperproc. And ulimit.
On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd
<robbie.sheph...@gmail.com> wrote:
>
> Does anyone know if there's a maximum number of open connections allowed when
> using hpricot/net::http?
>
> I have a bunch of xml feeds I need to read via hpricot and write to disk.
> Whenever I hit it too hard/fast, or when I try to open several files in
> hpricot in a loop, my app dies.
>
> If I do something like...
>
> arr = [url1, url2, url2]
> arr.each do|d|
> doc = Hpricot(open(d))
> doc.search(blah...)
> File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| f.write doc }
> end
>
> I get:
>
> OpenURI::HTTPError: 500 Internal Server Error from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in
> `open_http'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in
> `buffer_open'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in
> `open_loop'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in
> `catch'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in
> `open_loop'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in
> `open_uri'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in
> `open'
> from
> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in
> `open'
> from (irb):11
> from (irb):11:in `each'
> from (irb):11
>
> Am I missing something here, or is there a better way of doing this? I'm on
> Ruby 1.8.7 and hpricot 0.8.2
>
> Thanks
> Robbie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-ocea...@googlegroups.com.
> To unsubscribe from this group, send email to
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rails-oceania?hl=en.
--
http://crazyhollywood.org
--
You received this message because you are subscribed to the Google Groups "Ruby
or Rails Oceania" group.
To post to this group, send email to rails-ocea...@googlegroups.com.
To unsubscribe from this group, send email to
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rails-oceania?hl=en.
