Hi Could you try interacting directly with the server (i.e. without the proxy part?). We are not able to reproduce this.. The server makes a call equivalent to a onevm list and returns the output, in our installation with ~500 VMs we are seeing a 1s overhead because of the HTTP process. But the overall response time is < 2s for the EC2 interface and <1s from the CLI.
Cheers Ruben On Mon, Feb 6, 2012 at 11:42 AM, Ulrich Schwickerath <ulrich.schwicker...@cern.ch> wrote: > Hi, all, > > we are still experiencing problems after the upgrade to ONE3.2.1. The EC2 > access is very very slow and unstable. This morning, I noticed a lot of > blocked requests to econe, and error message in the econe-server log files: > > > NoMethodError - undefined method `[]' for nil:NilClass: > /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:216:in `render_state' > (erb):20:in `describe_instances' > /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `call' > /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `each_element' > /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:239:in > `each' > /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in > `upto' > /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in > `each' > /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:324:in `each_element' > /usr/lib/one/ruby/OpenNebula/Pool.rb:100:in `each' > (erb):14:in `describe_instances' > /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:181:in `describe_instances' > /usr/lib/one/ruby/cloud/econe/econe-server.rb:176:in `do_http_request' > /usr/lib/one/ruby/cloud/econe/econe-server.rb:158:in `POST /' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in > `compile!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in > `instance_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in > `route_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in > `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in > `process_route' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in > `catch' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in > `process_route' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in > `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in > `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in > `dispatch!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in > `call!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `instance_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `invoke' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `catch' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `invoke' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in > `call!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/head.rb:9:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/showexceptions.rb:21:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in > `synchronize' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in > `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in > `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in > `process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in > `receive_data' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in > `run_machine' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in > `run' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in > `start' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1234:in > `run!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/main.rb:25 > /usr/lib/one/ruby/cloud/econe/econe-server.rb:165 > NoMethodError - undefined method `[]' for nil:NilClass: > /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:216:in `render_state' > (erb):20:in `describe_instances' > /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `call' > /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `each_element' > /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:239:in > `each' > /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in > `upto' > /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in > `each' > /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:324:in `each_element' > /usr/lib/one/ruby/OpenNebula/Pool.rb:100:in `each' > (erb):14:in `describe_instances' > /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:181:in `describe_instances' > /usr/lib/one/ruby/cloud/econe/econe-server.rb:176:in `do_http_request' > /usr/lib/one/ruby/cloud/econe/econe-server.rb:158:in `POST /' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in > `compile!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in > `instance_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in > `route_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in > `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in > `process_route' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in > `catch' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in > `process_route' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in > `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in > `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in > `dispatch!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in > `call!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `instance_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `invoke' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `catch' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in > `invoke' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in > `call!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/head.rb:9:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/showexceptions.rb:21:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in > `synchronize' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in > `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in > `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in > `process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in > `receive_data' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in > `run_machine' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in > `run' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in > `start' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1234:in > `run!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/main.rb:25 > /usr/lib/one/ruby/cloud/econe/econe-server.rb:165 > > What I mean with slow is that to get a list of O(400) virtual machines the > system needs O(5-10) minutes while local queries with onevm list take ~2 > seconds. > > We are currently bitten badly by this problem because we use this to refill > our virtual batch farm. Is there a downgrade path ? > > Any idea ? > > Thanks, > Ulrich > > -- > -------------------------------------- > Dr. Ulrich Schwickerath > CERN IT/PES-PS > 1211 Geneva 23 > e-mail: ulrich.schwicker...@cern.ch > phone: +41 22 767 9576 > mobile: +41 76 487 5602 > > _______________________________________________ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- Ruben S. Montero, PhD Project co-Lead and Chief Architect OpenNebula - The Open Source Toolkit for Data Center Virtualization www.OpenNebula.org | rsmont...@opennebula.org | @OpenNebula _______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org