Hi,I found that the problem was caused by firewall between the machines. Here are the commands that helped me resolve the issue.
(works on Red Hat and Fedora) # /etc/init.d/iptables save # /etc/init.d/iptables stopThese clears all the rules from the firewall table. This made no route to host issues vanish.
Now the benchmark runs to completion but gives status as FAILED.I am getting a lot of these warning messages of connection refused in the run log.
UIDriverAgent[0].6.doHomePage: Connection refused Note: Error not counted in result. Either transaction start or end time is not within steady state.But did anyone got these warnings which led to failure status at the end of benchmark run? The results show that none of the operations actually succeeded. For all operations, the connection was refused.
Thanks, Jayneel On 07/06/2012 07:45 PM, Jayneel Gandhi wrote:
Hi,I was able to get rid hostnames from all places possible (places other than faban and all machines involved) and replaced them with ip addresses. The CmdService now atleast is getting setup on one of the machines (Backend+Client). Now it is failing on the other machine (Frontend).Time Host Level Message19:34:16 INFO Starting OlioDriver run using /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1L/19:34:16 INFO START TIME : Fri Jul 06 19:34:16 CDT 2012 19:34:16 INFO Starting Registry. 19:34:16 sc-h03 INFO Registry started. 19:34:26 WARNING/exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1L&exception=4>/ Agent(daemon)@128.105.102.201: No route to host. Will try remote shell instead. 19:34:26 WARNING [/usr/bin/ssh, -n, 128.105.102.201, /root/jayneel/cloudsuite/webserving/web-release/faban/bin/agent, 128.105.102.201, 128.105.102.203, 128.105.102.203, /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64, -Dfaban.home=/root/jayneel/cloudsuite/webserving/web-release/faban/, -Djava.security.policy=/root/jayneel/cloudsuite/webserving/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/root/jayneel/cloudsuite/webserving/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xmx3g, -Xms256m, -XX:+DisableExplicitGC, faban.benchmarkName=OlioDriver, faban.download=http://128.105.102.203:9980/]stdout: kernel.randomize_va_space = 019:34:46 INFO CmdService: Configured [email protected] on server sc-h0319:34:46 SEVERE/exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1L&exception=7>/ Error accessing command agent on system 128.105.102.20119:34:46 SEVERE CmdService setup failed. Exiting19:34:52 sc-h03 INFO [/root/jayneel/cloudsuite/webserving/web-release/faban/bin/fenxi, process, /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1L/, /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1L//post/, OlioDriver.1L]stdout: Creating database ... 3.80s Parsing raw files ... 0.00s Processing profile default_profile ...OK Opening Database ... 0.72s Executing queries ... 0.03s Generating html pages ... 0.11s 19:34:53 INFO Killing benchmark 19:34:53 INFO killing [email protected] 19:34:53 INFO killing [email protected] 19:34:53 SEVERE/exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1L&exception=13>/ Kill Failed for [email protected]19:34:54 INFO OlioDriver Completed/Terminated The exception saysjava.rmi.ConnectIOException: Exception creating connection to: 128.105.102.201; nested exception is:java.net.NoRouteToHostException: No route to hostBoth machines are able to connect/ssh each other without passwords. These machines are on the same network. I am not sure why the route not being found to the frontend machine.Jayneel On 07/06/2012 01:41 AM, [email protected] wrote:Yes, I have recompiled according to the step 11-17 on the setting up the faban driver from webpage. I am still getting these messages of not able to reconnect. 01:42:12 WARNING Retry connecting [email protected], count 1. 01:42:22 WARNING Retry connecting [email protected], count 2. 01:42:32 WARNING Retry connecting [email protected], count 3. 01:42:42 WARNING Retry connecting [email protected], count 4. 01:42:52 WARNING Retry connecting [email protected], count 5. 01:43:02 WARNING Retry connecting [email protected], count 6. 01:43:12 WARNING Retry connecting [email protected], count 7. 01:43:22 WARNING Retry connecting [email protected], count 8. 01:43:32 WARNING Retry connecting [email protected], count 9. 01:43:42 WARNING Retry connecting [email protected], count 10. 01:43:42 SEVERE Could not connect [email protected] 01:43:42 SEVERE CmdService setup failed. Exiting JayneelHave you recompile Faban following steps 11 to 17 from "Setting up the Faban driver" installation guide? Leonardo On Fri, Jul 6, 2012 at 2:24 AM,<[email protected]> wrote:Hi Binzhang, I did change the hostnames to ip-address. It was still giving the [email protected] connection retries. I did see something peculiar in the message in the run after changing all reference of hostnames to ip-addresses. The java call stack printed in the end after the benchmark is killed still has reference to sc-h03. Do I need to do something specific in faban except from restart? Jayneel Killing benchmark 00:23:10 WARNING [/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java, -Dfaban.home=/root/jayneel/cloudsuite/webserving/web-release/faban/, -Djava.security.policy=/root/jayneel/cloudsuite/webserving/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/root/jayneel/cloudsuite/webserving/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xms32m, -Xmx1024m, -cp, /root/jayneel/cloudsuite/webserving/web-release/faban/lib/commons-codec-1.2.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/commons-httpclient-3.1.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/commons-logging.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/fabancommon.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/fabandriver.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/fabanagents.jar, com.sun.faban.common.RegistryImpl] stderr: SocketHandler: connect failed to sc-h03:9999 Can't load log handler "java.util.logging.SocketHandler" java.net.UnknownHostException: sc-h03 java.net.UnknownHostException: sc-h03 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:158) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) at java.net.Socket.connect(Socket.java:546) at java.net.Socket.connect(Socket.java:495) at java.net.Socket.<init>(Socket.java:392) at java.net.Socket.<init>(Socket.java:206) at java.util.logging.SocketHandler.connect(SocketHandler.java:154) at java.util.logging.SocketHandler.<init>(SocketHandler.java:112) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532) at java.lang.Class.newInstance0(Class.java:372) at java.lang.Class.newInstance(Class.java:325) at java.util.logging.LogManager$3.run(LogManager.java:384) at java.security.AccessController.doPrivileged(Native Method) at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:370) at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:906) at java.util.logging.LogManager.access$900(LogManager.java:147) at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:987) at java.util.logging.Logger.log(Logger.java:472) at java.util.logging.Logger.doLog(Logger.java:498) at java.util.logging.Logger.log(Logger.java:521) at java.util.logging.Logger.info(Logger.java:1040) at com.sun.faban.common.RegistryImpl.main(RegistryImpl.java:427)I met the same problem and solved it by using the ip rather than the hostname of the remote machine, say the "sc-h03.cs.wisc.edu". It was weired since i added the entry in hosts file, and can ping and ssh correctly with the hostname. I don't know the real reason, but it worked. Maybe you could try it. Regards, Binzhang Fu äºZ( 2012/7/6 3:25, Jayneel Gandhi å+^(TM)é?":Hi Cansu, Both machines are able to connect with ssh without password. The log that I sent is after this setup was done. Jayneel On Jul 5, 2012, at 1:16 PM, Cansu Kaynak<[email protected]> wrote:Please make sure that all machines can ssh to each other and then try starting the experiment. Cansu On 07/05/2012 08:05 PM, Jayneel Gandhi wrote:Hi Cansu, Thanks for the suggestion. The problem was not a field left empty but one of the address provided was wrong in the configuration. I corrected that. But it lead to another issue. This was the log. Time Host Level Message 12:05:32 INFO Starting OlioDriver run using /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1E/ 12:05:32 INFO START TIME : Thu Jul 05 12:05:32 CDT 2012 12:05:32 INFO Starting Registry. 12:05:42 WARNING/ exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1E&exception=3>/ Agent(daemon)@sc-h01.cs.wisc.edu: No route to host. Will try remote shell instead. 12:05:48 WARNING [/usr/bin/ssh, -n, sc-h01.cs.wisc.edu, /root/jayneel/cloudsuite/webserving/web-release/faban/bin/agent, sc-h01.cs.wisc.edu, sc-h03.cs.wisc.edu, 128.105.102.203, /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64, -Dfaban.home=/root/jayneel/cloudsuite/webserving/web-release/faban/, -Djava.security.policy=/root/jayneel/cloudsuite/webserving/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/root/jayneel/cloudsuite/webserving/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xmx3g, -Xms256m, -XX:+DisableExplicitGC, faban.benchmarkName=OlioDriver, faban.download=http://sc-h03.cs.wisc.edu:9980/] stdout: kernel.randomize_va_space = 0 12:06:12 WARNING Retry connecting to [email protected], count 1. 12:06:22 WARNING Retry connecting to [email protected], count 2. 12:06:32 WARNING Retry connecting to [email protected], count 3. 12:06:42 WARNING Retry connecting to [email protected], count 4. 12:06:52 WARNING Retry connecting to [email protected], count 5. 12:07:02 WARNING Retry connecting to [email protected], count 6. 12:07:12 WARNING Retry connecting to [email protected], count 7. 12:07:22 WARNING Retry connecting to [email protected], count 8. 12:07:32 WARNING Retry connecting to [email protected], count 9. 12:07:42 WARNING Retry connecting to [email protected], count 10. 12:07:42 SEVERE Could not connect to [email protected] 12:07:42 SEVERE CmdService setup failed. Exiting 12:07:42 SEVERE/ exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1E&exception=17>/ FenXi process command [fenxi, process, /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1E/, /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1E//post/, OlioDriver.1E] failed. 12:07:42 INFO Killing benchmark 12:07:42 WARNING [/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java, -Dfaban.home=/root/jayneel/cloudsuite/webserving/web-release/faban/, -Djava.security.policy=/root/jayneel/cloudsuite/webserving/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/root/jayneel/cloudsuite/webserving/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xms32m, -Xmx1024m, -cp, /root/jayneel/cloudsuite/webserving/web-release/faban/lib/commons-codec-1.2.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/commons-httpclient-3.1.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/commons-logging.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/fabancommon.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/fabandriver.jar:/root/jayneel/cloudsuite/webserving/web-release/faban/lib/fabanagents.jar, com.sun.faban.common.RegistryImpl] stderr: SocketHandler: connect failed to sc-h03:9999 Can't load log handler "java.util.logging.SocketHandler" java.net.UnknownHostException: sc-h03 java.net.UnknownHostException: sc-h03 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:158) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) at java.net.Socket.connect(Socket.java:546) at java.net.Socket.connect(Socket.java:495) at java.net.Socket.<init>(Socket.java:392) at java.net.Socket.<init>(Socket.java:206) at java.util.logging.SocketHandler.connect(SocketHandler.java:154) at java.util.logging.SocketHandler.<init>(SocketHandler.java:112) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532) at java.lang.Class.newInstance0(Class.java:372) at java.lang.Class.newInstance(Class.java:325) at java.util.logging.LogManager$3.run(LogManager.java:384) at java.security.AccessController.doPrivileged(Native Method) at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:370) at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:906) at java.util.logging.LogManager.access$900(LogManager.java:147) at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:987) at java.util.logging.Logger.log(Logger.java:472) at java.util.logging.Logger.doLog(Logger.java:498) at java.util.logging.Logger.log(Logger.java:521) at java.util.logging.Logger.info(Logger.java:1040) at com.sun.faban.common.RegistryImpl.main(RegistryImpl.java:427) ------------------------------------------------------------------------ I am not sure why the connection to sc-h03 is not happening. This is the run configuration. /usr/lib/jvm/java-Xmx3g -Xms256m -XX:+DisableExplicitGCEnter description for this run heresc-h03.cs.wisc.edutrue0vmstat 10253030010/root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1E/false100falsefalse63025130phpsc-h01.cs.wisc.edusc-h01.cs.wisc.edu:8080true0vmstat 10; mpstat 10; nicstat 10; iostat -x 10ApacheHttpdServiceNONEtrue/usr/local/nginx/logs/tmp/http_sessions/usr/local/nginx/conf/nginx.conf/root/jayneel/cloudsuite/webserving/web-release/app_dir/etc/php.inisc-h03.cs.wisc.edutrue0com.mysql.jdbc.Driversc-h03.cs.wisc.edu:mysql://dbHost/olio?user=olio&password=olio&relaxAutoCommit=true&sessionVariables=FOREIGN_KEY_CHECKS=0truefalse25MySQLServiceMysqlstatsfalse/var/mysqlolioolio/etc/my.cnfsc-h01.cs.wisc.edutrue0NONEfalse/root/jayneel/cloudsuite/webserving/web-release/filestoretrue0NONEMemcachedServiceMemcacheStats -i 10true/usr/lib/memcached256 Both the machines are in the same network and can connect to each other. Do have some suggestion? Thanks, Jayneel On 07/04/2012 05:28 PM, Cansu Kaynak wrote:It seems like a configuration field was left empty or has an inappropriate value while setting up the test run. You can find the details about the cause of the failure if you click on the "exception" links on the page you posted. After making sure that you filled out all the fields as described in the instructions, if you still get the same error, you can send your configuration parameters. -- Cansu On Jul 5, 2012, at 12:17 AM, Jayneel Gandhi wrote:Hi, I installed the web serving benchmark as specified on the cloudsuite webpage and tried to perform a benchmark run. The run failed saying "Benchmark Validation Failed". The stack providing some java thread stack. Time Host Level Message 17:16:26 INFO Starting OlioDriver run using /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1A/ 17:16:26 INFO START TIME : Wed Jul 04 17:16:26 CDT 2012 17:16:26 SEVERE/ exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1A&exception=2>/ Benchmark validation failed. 17:16:26 SEVERE/ exception <http://sc-h03.cs.wisc.edu:9980/LogReader?runId=OlioDriver.1A&exception=3>/ FenXi process command [fenxi, process, /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1A/, /root/jayneel/cloudsuite/webserving/web-release/faban/output/OlioDriver.1A//post/, OlioDriver.1A] failed. 17:16:26 INFO Killing benchmark 17:16:26 INFO OlioDriver Completed/Terminated Do you guys have have some pointers that I can look into to fix the issue? Thanks, Jayneel
