you may well be right Jonathan. I took Marks idea and used a standard net:http response to write out the contents to disk instead of using hpricot. I was then able to look at the response's status code and could see 500 errors being returned occassionally from the remote server.
Thanks to everyone for their suggestions and help. On Wed, Jul 28, 2010 at 11:05 AM, Jonathan Clarke <jonat...@beilabs.com>wrote: > To me that looks like the remote servers issue, you could be hammering > it, put a sleep command in to let it recover a little. > > Jonathan > > On 28 July 2010 10:41, Robbie Shepherd <robbie.sheph...@gmail.com> wrote: > > Its a rails app, deployed with passenger and apache, but I get the same > > issues when running it locally on os x as well. > > > > Mark - that's a good idea, will give that a go, see if I can bypass > hpricot > > for getting the response. > > > > I'll try and get a better stack trace, will reply in a bit. Thanks for > the > > suggestions so far. > > > > Robbie > > > > > > On Wed, Jul 28, 2010 at 10:37 AM, Lachie <lach...@gmail.com> wrote: > >> > >> Its the app at the other end with the problem "500 Internal Server > Error" > >> > >> Is that within your control too? Can you give us more details on that > >> app, and how its deployed? > >> > >> e.g. is it a rails app deployed with passenger behind apache? > >> > >> :lachie > >> http://plus2.com.au > >> http://smartbomb.com.au > >> http://www.flickr.com/photos/lachie/ > >> > >> > >> > >> On Wed, Jul 28, 2010 at 10:29 AM, Robbie Shepherd > >> <robbie.sheph...@gmail.com> wrote: > >> > Simon, yes they are. > >> > > >> > Julio, I'm getting the same error on both os x and my server running > >> > Ubuntu > >> > > >> > Robbie > >> > > >> > On Wed, Jul 28, 2010 at 10:28 AM, Simon Russell <si...@bellyphant.com > > > >> > wrote: > >> >> > >> >> Are they all coming from the same server? > >> >> > >> >> On Wed, Jul 28, 2010 at 10:25, Mark Wotton <mwot...@gmail.com> > wrote: > >> >> > On three URLs? Surely not... > >> >> > > >> >> > mark > >> >> > > >> >> > On Wed, Jul 28, 2010 at 10:24 AM, Julio Cesar Ody > >> >> > <julio...@gmail.com> > >> >> > wrote: > >> >> >> I can't really tell from that stacktrace if the problem is this, > but > >> >> >> you may be hitting the maximum number of open files (and sockets). > >> >> >> > >> >> >> Are you on OSX? If so, you can expand that limit with sysctl. Try: > >> >> >> > >> >> >> sudo sysctl -w kern.maxfiles=36000 > >> >> >> > >> >> >> Sets to a bit over twice as much the default. Also, check > >> >> >> kern.maxfilesperproc. And ulimit. > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd > >> >> >> <robbie.sheph...@gmail.com> wrote: > >> >> >>> > >> >> >>> Does anyone know if there's a maximum number of open connections > >> >> >>> allowed when using hpricot/net::http? > >> >> >>> > >> >> >>> I have a bunch of xml feeds I need to read via hpricot and write > to > >> >> >>> disk. Whenever I hit it too hard/fast, or when I try to open > >> >> >>> several files > >> >> >>> in hpricot in a loop, my app dies. > >> >> >>> > >> >> >>> If I do something like... > >> >> >>> > >> >> >>> arr = [url1, url2, url2] > >> >> >>> arr.each do|d| > >> >> >>> doc = Hpricot(open(d)) > >> >> >>> doc.search(blah...) > >> >> >>> File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| > f.write > >> >> >>> doc } > >> >> >>> end > >> >> >>> > >> >> >>> I get: > >> >> >>> > >> >> >>> OpenURI::HTTPError: 500 Internal Server Error from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in > >> >> >>> `open_http' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in > >> >> >>> `buffer_open' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in > >> >> >>> `open_loop' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in > >> >> >>> `catch' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in > >> >> >>> `open_loop' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in > >> >> >>> `open_uri' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in > >> >> >>> `open' > >> >> >>> from > >> >> >>> > >> >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in > >> >> >>> `open' > >> >> >>> from (irb):11 > >> >> >>> from (irb):11:in `each' > >> >> >>> from (irb):11 > >> >> >>> > >> >> >>> Am I missing something here, or is there a better way of doing > >> >> >>> this? > >> >> >>> I'm on Ruby 1.8.7 and hpricot 0.8.2 > >> >> >>> > >> >> >>> Thanks > >> >> >>> Robbie > >> >> >>> > >> >> >>> -- > >> >> >>> You received this message because you are subscribed to the > Google > >> >> >>> Groups "Ruby or Rails Oceania" group. > >> >> >>> To post to this group, send email to > >> >> >>> rails-ocea...@googlegroups.com. > >> >> >>> To unsubscribe from this group, send email to > >> >> >>> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> >> >>> For more options, visit this group at > >> >> >>> http://groups.google.com/group/rails-oceania?hl=en. > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> http://crazyhollywood.org > >> >> >> > >> >> >> -- > >> >> >> You received this message because you are subscribed to the Google > >> >> >> Groups "Ruby or Rails Oceania" group. > >> >> >> To post to this group, send email to > rails-ocea...@googlegroups.com. > >> >> >> To unsubscribe from this group, send email to > >> >> >> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> >> >> For more options, visit this group at > >> >> >> http://groups.google.com/group/rails-oceania?hl=en. > >> >> >> > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > A UNIX signature isn't a return address, it's the ASCII equivalent > of > >> >> > a > >> >> > black velvet clown painting. It's a rectangle of carets surrounding > a > >> >> > quote from a literary giant of weeniedom like Heinlein or Dr. Who. > >> >> > -- Chris Maeda > >> >> > > >> >> > -- > >> >> > You received this message because you are subscribed to the Google > >> >> > Groups "Ruby or Rails Oceania" group. > >> >> > To post to this group, send email to > rails-ocea...@googlegroups.com. > >> >> > To unsubscribe from this group, send email to > >> >> > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> >> > For more options, visit this group at > >> >> > http://groups.google.com/group/rails-oceania?hl=en. > >> >> > > >> >> > > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > >> >> Groups > >> >> "Ruby or Rails Oceania" group. > >> >> To post to this group, send email to rails-ocea...@googlegroups.com. > >> >> To unsubscribe from this group, send email to > >> >> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> >> For more options, visit this group at > >> >> http://groups.google.com/group/rails-oceania?hl=en. > >> >> > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Ruby or Rails Oceania" group. > >> > To post to this group, send email to rails-ocea...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> > For more options, visit this group at > >> > http://groups.google.com/group/rails-oceania?hl=en. > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Ruby or Rails Oceania" group. > >> To post to this group, send email to rails-ocea...@googlegroups.com. > >> To unsubscribe from this group, send email to > >> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> For more options, visit this group at > >> http://groups.google.com/group/rails-oceania?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Ruby or Rails Oceania" group. > > To post to this group, send email to rails-ocea...@googlegroups.com. > > To unsubscribe from this group, send email to > > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > > For more options, visit this group at > > http://groups.google.com/group/rails-oceania?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to rails-ocea...@googlegroups.com. > To unsubscribe from this group, send email to > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to rails-ocea...@googlegroups.com. To unsubscribe from this group, send email to rails-oceania+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
