Its a rails app, deployed with passenger and apache, but I get the same issues when running it locally on os x as well.
Mark - that's a good idea, will give that a go, see if I can bypass hpricot for getting the response. I'll try and get a better stack trace, will reply in a bit. Thanks for the suggestions so far. Robbie On Wed, Jul 28, 2010 at 10:37 AM, Lachie <lach...@gmail.com> wrote: > Its the app at the other end with the problem "500 Internal Server Error" > > Is that within your control too? Can you give us more details on that > app, and how its deployed? > > e.g. is it a rails app deployed with passenger behind apache? > > :lachie > http://plus2.com.au > http://smartbomb.com.au > http://www.flickr.com/photos/lachie/ > > > > On Wed, Jul 28, 2010 at 10:29 AM, Robbie Shepherd > <robbie.sheph...@gmail.com> wrote: > > Simon, yes they are. > > > > Julio, I'm getting the same error on both os x and my server running > Ubuntu > > > > Robbie > > > > On Wed, Jul 28, 2010 at 10:28 AM, Simon Russell <si...@bellyphant.com> > > wrote: > >> > >> Are they all coming from the same server? > >> > >> On Wed, Jul 28, 2010 at 10:25, Mark Wotton <mwot...@gmail.com> wrote: > >> > On three URLs? Surely not... > >> > > >> > mark > >> > > >> > On Wed, Jul 28, 2010 at 10:24 AM, Julio Cesar Ody <julio...@gmail.com > > > >> > wrote: > >> >> I can't really tell from that stacktrace if the problem is this, but > >> >> you may be hitting the maximum number of open files (and sockets). > >> >> > >> >> Are you on OSX? If so, you can expand that limit with sysctl. Try: > >> >> > >> >> sudo sysctl -w kern.maxfiles=36000 > >> >> > >> >> Sets to a bit over twice as much the default. Also, check > >> >> kern.maxfilesperproc. And ulimit. > >> >> > >> >> > >> >> > >> >> > >> >> On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd > >> >> <robbie.sheph...@gmail.com> wrote: > >> >>> > >> >>> Does anyone know if there's a maximum number of open connections > >> >>> allowed when using hpricot/net::http? > >> >>> > >> >>> I have a bunch of xml feeds I need to read via hpricot and write to > >> >>> disk. Whenever I hit it too hard/fast, or when I try to open several > files > >> >>> in hpricot in a loop, my app dies. > >> >>> > >> >>> If I do something like... > >> >>> > >> >>> arr = [url1, url2, url2] > >> >>> arr.each do|d| > >> >>> doc = Hpricot(open(d)) > >> >>> doc.search(blah...) > >> >>> File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| f.write > >> >>> doc } > >> >>> end > >> >>> > >> >>> I get: > >> >>> > >> >>> OpenURI::HTTPError: 500 Internal Server Error from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in > >> >>> `open_http' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in > >> >>> `buffer_open' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in > >> >>> `open_loop' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in > >> >>> `catch' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in > >> >>> `open_loop' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in > >> >>> `open_uri' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in > >> >>> `open' > >> >>> from > >> >>> > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in > >> >>> `open' > >> >>> from (irb):11 > >> >>> from (irb):11:in `each' > >> >>> from (irb):11 > >> >>> > >> >>> Am I missing something here, or is there a better way of doing this? > >> >>> I'm on Ruby 1.8.7 and hpricot 0.8.2 > >> >>> > >> >>> Thanks > >> >>> Robbie > >> >>> > >> >>> -- > >> >>> You received this message because you are subscribed to the Google > >> >>> Groups "Ruby or Rails Oceania" group. > >> >>> To post to this group, send email to rails-oceania@googlegroups.com > . > >> >>> To unsubscribe from this group, send email to > >> >>> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> >>> For more options, visit this group at > >> >>> http://groups.google.com/group/rails-oceania?hl=en. > >> >> > >> >> > >> >> > >> >> -- > >> >> http://crazyhollywood.org > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > >> >> Groups "Ruby or Rails Oceania" group. > >> >> To post to this group, send email to rails-ocea...@googlegroups.com. > >> >> To unsubscribe from this group, send email to > >> >> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> >> For more options, visit this group at > >> >> http://groups.google.com/group/rails-oceania?hl=en. > >> >> > >> >> > >> > > >> > > >> > > >> > -- > >> > A UNIX signature isn't a return address, it's the ASCII equivalent of > a > >> > black velvet clown painting. It's a rectangle of carets surrounding a > >> > quote from a literary giant of weeniedom like Heinlein or Dr. Who. > >> > -- Chris Maeda > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "Ruby or Rails Oceania" group. > >> > To post to this group, send email to rails-ocea...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> > For more options, visit this group at > >> > http://groups.google.com/group/rails-oceania?hl=en. > >> > > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Ruby or Rails Oceania" group. > >> To post to this group, send email to rails-ocea...@googlegroups.com. > >> To unsubscribe from this group, send email to > >> rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > >> For more options, visit this group at > >> http://groups.google.com/group/rails-oceania?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Ruby or Rails Oceania" group. > > To post to this group, send email to rails-ocea...@googlegroups.com. > > To unsubscribe from this group, send email to > > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > > For more options, visit this group at > > http://groups.google.com/group/rails-oceania?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to rails-ocea...@googlegroups.com. > To unsubscribe from this group, send email to > rails-oceania+unsubscr...@googlegroups.com<rails-oceania%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to rails-ocea...@googlegroups.com. To unsubscribe from this group, send email to rails-oceania+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
