Cansu, Thanks alot for your reply.
One more question before I stop bugging. For the websearch workload, http://parsa.epfl.ch/cloudsuite/search.html says at the end: The statistics are kept in the outputDir/number_of_run/summary.xml. Among other the following statistics are kept: <metric unit=â€ops/secâ€>27.760</metric> <users>55</users> <responseTimes unit="seconds"> <operation name="GET" r90th="0.500"> <avg>0.101</avg> <max>1.095</max> <sd>0.116</sd> <p90th>0.250</p90th> <passed>true</passed> <p99th>0.560</p99th> </operation> </responseTimes> Question: 1) Is outputDir/number_of_run/summary.xml referring to statistics being kept in the simulated machine or in simpflex ? Expanding on that, are these values changeable in simflex or does the xml have to be loaded into the simulated machine ? 2) Are these reported values (90th percentile, 99th percentile etc.) meaningful only in a native setting where the benchmarks are entirely run on native machines ? If not, do they accurately reflect the 90th and 99th percentile values in a simflex timing simulation ? Thanks Hamza ----- Original Message ----- From: "Cansu Kaynak" <[email protected]> To: "Hamza Bin Sohail" <[email protected]> Cc: [email protected] Sent: Sunday, June 29, 2014 1:28:21 PM Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping Dear Hamza, 1) Technically, it is possible to insert magic instructions for each workload to count the number of transactions completed during the simulation. However, because Flexus uses statistical sampling to speed up cycle-accurate simulations of representative execution windows in server workloads (which can be on the order of tens of seconds), we use user IPC to measure performance. User IPC has been shown to accurately reflect overall system throughput. For more information about statistical sampling and use of user IPC, you can take a look at the IEEE Micro paper called SimFlex: Statistical Sampling of Computer System Simulation ( http://infoscience.epfl.ch/record/112669/files/simflex.pdf?version=1 ). 2) Yes, there is a separate machine for each tier in multi-tier workload setups. 3) We collect the user IPC of each flexpoint (sampling unit) and then calculate the average user IPC along with an average error for a given confidence level as described in http://parsa.epfl.ch/cloudsuite/SMARTS.pdf . Let us know if you have more questions. -- Cansu On 29 Jun 2014, at 18:55, Hamza Bin Sohail < [email protected] > wrote: ping ----- Original Message ----- From: "Hamza Bin Sohail" < [email protected] > To: [email protected] Sent: Thursday, June 26, 2014 9:10:06 PM Subject: Questions regarding cloudsuite request-oriented benchmarks Hi all, I wanted to ask a few questions regarding request-oriented scaleout workloads in cloudsuite e.g memcached, websearch etc. I'm interested in downloading the simics images but before that, I had a few questions. 1) In the Wisconsin GEMS simulation infrastructure, there is a clear notion of transactions for commercial workloads and performance is measured in terms of throughput i.e cycles per transaction , and not in terms of instructions executed. The reason obviously being that it is perfectly possible for multi-threaded workloads to be executing instructions while not doing any useful work e.g busy-waiting etc. Whenever a transaction is initiated or ended, the drivers inform the simulator (simics) through magic instructions. The magic instructions are event notification mechanisms. So the question is: Do the cloudsuite simics images for memcached,websearch and other request-oriented workloads have that mechanism in place to inform simflex about the completion of events ? For example, when the Faban driver gets a query response in websearch, or initiates one, does it tell Simflex about it through magic instructions ? If not, what mechanism does simflex use to report the number of requests completed in the final reported out ? 2) I am guessing these workloads setup as a distributed simics simulation, correct ? For example, for websearch, is the frontend server on a separate simulated machine, and the driver on a separate simulated machine, and the index nodes on a separate simulated machine ? If yes, and if the answer to my question in 1) is yes as well, then for websearch, are the magic instructions placed in the faban driver to notify whether a request has ended or started ? 3) This question is only relevant if simflex does not know when a request was generated or completed. If the answer to 1) is no, how is performance measured ? Is the benchmark run for fixed number of instructions in each sampling phase and then their execution times in each phase is aggregated / extrapolated to what the most likely execution time would've been if the entire benchmark were run to completion ? Thanks alot Hamza
