Re: Questions regarding cloudsuite request-oriented benchmarks - ping

[javier . picorel]

G,,,,xXv Sent from my BlackBerry 10 smartphone. From: Volos Stavros Sent: Tuesday, 15 July 2014 13:29 To: ‎Hamza Bin Sohail Cc: <cloudsuite@listes.epfl.ch> Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping Hi Hamza, The statistics correspond to the emulated machine (as modelled by Simics). You cannot use those files to perform an QoS analysis for the Flexus-simulated CMP. In case you want to perform a QoS analysis, the benchmarks include magic breakpoints at various phases. Collecting the magic breakpoints of each individual request and the associated response, you can look at the QoS metrics. Regards, -Stavros. On Jul 5, 2014, at 6:46 AM, Hamza Bin Sohail <hsoh...@purdue.edu> wrote: > > ping > > ----- Original Message ----- > From: "Hamza Bin Sohail" <hsoh...@purdue.edu> > To: "Cansu Kaynak" <cansu.kay...@epfl.ch> > Cc: cloudsuite@listes.epfl.ch > Sent: Thursday, July 3, 2014 1:22:00 AM > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks > > Cansu, > > Thanks alot for your reply. > > One more question before I stop bugging. > > For the websearch workload, http://parsa.epfl.ch/cloudsuite/search.html > says at the end: > > The statistics are kept in the outputDir/number_of_run/summary.xml. Among other the following statistics are kept: <metric unit=”ops/sec”>27.760</metric> > <users>55</users> > <responseTimes unit="seconds"> > <operation name="GET" r90th="0.500"> > <avg>0.101</avg> > <max>1.095</max> > <sd>0.116</sd> > <p90th>0.250</p90th> > <passed>true</passed> > <p99th>0.560</p99th> > </operation> > </responseTimes> > > Question: > 1) Is outputDir/number_of_run/summary.xml referring to statistics being > kept in the simulated machine or in simpflex ? Expanding on that, are these values changeable in simflex or does the xml have to be loaded into the simulated machine ? > > 2) Are these reported values (90th percentile, 99th percentile etc.) meaningful only in a native setting where the benchmarks are entirely run on native machines ? If not, do they accurately reflect the 90th and 99th percentile values in a simflex timing simulation ? > > > Thanks > > Hamza > > ----- Original Message ----- > From: "Cansu Kaynak" <cansu.kay...@epfl.ch> > To: "Hamza Bin Sohail" <hsoh...@purdue.edu> > Cc: cloudsuite@listes.epfl.ch > Sent: Sunday, June 29, 2014 1:28:21 PM > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping > > Dear Hamza, > > > 1) Technically, it is possible to insert magic instructions for each workload to count the number of transactions completed during the simulation. > However, because Flexus uses statistical sampling to speed up cycle-accurate simulations of representative execution windows in server workloads (which can be on the order of tens of seconds), we use user IPC to measure performance. > User IPC has been shown to accurately reflect overall system throughput. > For more information about statistical sampling and use of user IPC, you can take a look at the IEEE Micro paper called SimFlex: Statistical Sampling of Computer System Simulation ( http://infoscience.epfl.ch/record/112669/files/simflex.pdf?version=1 ). > > > 2) Yes, there is a separate machine for each tier in multi-tier workload setups. > > > 3) We collect the user IPC of each flexpoint (sampling unit) and then calculate the average user IPC along with an average error for a given confidence level as described in http://parsa.epfl.ch/cloudsuite/SMARTS.pdf . > > > Let us know if you have more questions. > > > > -- > Cansu > > > > On 29 Jun 2014, at 18:55, Hamza Bin Sohail < hsoh...@purdue.edu > wrote: > > > ping > > ----- Original Message ----- > From: "Hamza Bin Sohail" < hsoh...@purdue.edu > > To: cloudsuite@listes.epfl.ch > Sent: Thursday, June 26, 2014 9:10:06 PM > Subject: Questions regarding cloudsuite request-oriented benchmarks > > > Hi all, > > I wanted to ask a few questions regarding request-oriented scaleout workloads in cloudsuite e.g memcached, websearch etc. I'm interested in downloading the simics images but before that, I had a few questions. > > 1) In the Wisconsin GEMS simulation infrastructure, there is a clear notion of transactions for commercial workloads and performance > is measured in terms of throughput i.e cycles per transaction , and not in terms of instructions executed. The reason obviously being that it is perfectly possible for multi-threaded workloads to be executing instructions while not doing any useful work e.g busy-waiting etc. Whenever a transaction is initiated or ended, the drivers inform the simulator (simics) through magic instructions. The magic instructions are event notification mechanisms. So the question is: Do the cloudsuite simics images for memcached,websearch and other request-oriented workloads have that mechanism in place to inform simflex about the completion of events ? For example, when the Faban driver gets a query response in websearch, or initiates one, does it tell Simflex about it through magic instructions ? If not, what mechanism does simflex use to report the number of requests completed in the final reported out ? > > 2) I am guessing these workloads setup as a distributed simics simulation, correct ? For example, for websearch, is the frontend server on a separate simulated machine, and the driver on a separate simulated machine, and the index nodes on a separate simulated machine ? If yes, and if the answer to my question in 1) is yes as well, then for websearch, are the magic instructions placed in the faban driver to notify whether a request has ended or started ? > > 3) This question is only relevant if simflex does not know when a request was generated or completed. If the answer to 1) is no, how is performance measured ? Is the benchmark run for fixed number of instructions in each sampling phase and then their execution times in each phase is aggregated / extrapolated to what the most likely execution time would've been if the entire benchmark were run to completion ? > > Thanks alot > > Hamza