Hi all
I am intersted on the problem of statistics QoS metrics on flexus. And I am wondering dose it necessary to recreate image and add magic instruction by myself? Could anyone gives me some detailed guides on how to use magic breakpoint to get QoS statistics? Thanks! Jing > ----- Original Message ----- > From: "Hamza Bin Sohail" <[email protected]> > > Cc: [email protected] > Sent: Thursday, July 3, 2014 1:22:00 AM > > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks Stavros, Thank you for your response. Just one clarification: I was under the impression, as Cansu told in another email thread, that the websearch images currently, do not have magic breakpoints in place when a request is made or completed. Are you saying that the magic instructions to mark these events are there in, say the websearch image ( which is downloadable from EPFL) and I can measure QoS metrics by just focussing on those events ? In that case, that would be great. Or are you saying that I can measure QoS metrics by recreating the images, and adding magic instructions myself when the request is initiated/ended?Thanks again. Hamza ----- Original Message ----- From: "Volos Stavros" <[email protected]> To: "Hamza Bin Sohail" <[email protected]> Cc: "<[email protected]>" <[email protected]> Sent: Tuesday, July 15, 2014 4:29:13 AM Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping Hi Hamza, The statistics correspond to the emulated machine (as modelled by Simics). You cannot use those files to perform an QoS analysis for the Flexus-simulated CMP. In case you want to perform a QoS analysis, the benchmarks include magic breakpoints at various phases. Collecting the magic breakpoints of each individual request and the associated response, you can look at the QoS metrics. Regards, -Stavros. On Jul 5, 2014, at 6:46 AM, Hamza Bin Sohail <[email protected]> wrote: > > ping > > ----- Original Message ----- > From: "Hamza Bin Sohail" <[email protected]> > To: "Cansu Kaynak" <[email protected]> > Cc: [email protected] > Sent: Thursday, July 3, 2014 1:22:00 AM > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks > > Cansu, > > Thanks alot for your reply. > > One more question before I stop bugging. > > For the websearch workload, http://parsa.epfl.ch/cloudsuite/search.html > says at the end: > > The statistics are kept in the outputDir/number_of_run/summary.xml. Among > other the following statistics are kept: <metric > unit=â€ops/secâ€>27.760</metric> > <users>55</users> > <responseTimes unit="seconds"> > <operation name="GET" r90th="0.500"> > <avg>0.101</avg> > <max>1.095</max> > <sd>0.116</sd> > <p90th>0.250</p90th> > <passed>true</passed> > <p99th>0.560</p99th> > </operation> > </responseTimes> > > Question: > 1) Is outputDir/number_of_run/summary.xml referring to statistics being > kept in the simulated machine or in simpflex ? Expanding on that, are these > values changeable in simflex or does the xml have to be loaded into the > simulated machine ? > > 2) Are these reported values (90th percentile, 99th percentile etc.) > meaningful only in a native setting where the benchmarks are entirely run on > native machines ? If not, do they accurately reflect the 90th and 99th > percentile values in a simflex timing simulation ? > > > Thanks > > Hamza > > ----- Original Message ----- > From: "Cansu Kaynak" <[email protected]> > To: "Hamza Bin Sohail" <[email protected]> > Cc: [email protected] > Sent: Sunday, June 29, 2014 1:28:21 PM > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping > > Dear Hamza, > > > 1) Technically, it is possible to insert magic instructions for each workload > to count the number of transactions completed during the simulation. > However, because Flexus uses statistical sampling to speed up cycle-accurate > simulations of representative execution windows in server workloads (which > can be on the order of tens of seconds), we use user IPC to measure > performance. > User IPC has been shown to accurately reflect overall system throughput. > For more information about statistical sampling and use of user IPC, you can > take a look at the IEEE Micro paper called SimFlex: Statistical Sampling of > Computer System Simulation ( > http://infoscience.epfl.ch/record/112669/files/simflex.pdf?version=1 ). > > > 2) Yes, there is a separate machine for each tier in multi-tier workload > setups. > > > 3) We collect the user IPC of each flexpoint (sampling unit) and then > calculate the average user IPC along with an average error for a given > confidence level as described in http://parsa.epfl.ch/cloudsuite/SMARTS.pdf . > > > Let us know if you have more questions. > > > > -- > Cansu > > > > On 29 Jun 2014, at 18:55, Hamza Bin Sohail < [email protected] > wrote: > > > ping > > ----- Original Message ----- > From: "Hamza Bin Sohail" < [email protected] > > To: [email protected] > Sent: Thursday, June 26, 2014 9:10:06 PM > Subject: Questions regarding cloudsuite request-oriented benchmarks > > > Hi all, > > I wanted to ask a few questions regarding request-oriented scaleout workloads > in cloudsuite e.g memcached, websearch etc. I'm interested in downloading the > simics images but before that, I had a few questions. > > 1) In the Wisconsin GEMS simulation infrastructure, there is a clear notion > of transactions for commercial workloads and performance > is measured in terms of throughput i.e cycles per transaction , and not in > terms of instructions executed. The reason obviously being that it is > perfectly possible for multi-threaded workloads to be executing instructions > while not doing any useful work e.g busy-waiting etc. Whenever a transaction > is initiated or ended, the drivers inform the simulator (simics) through > magic instructions. The magic instructions are event notification mechanisms. > So the question is: Do the cloudsuite simics images for memcached,websearch > and other request-oriented workloads have that mechanism in place to inform > simflex about the completion of events ? For example, when the Faban driver > gets a query response in websearch, or initiates one, does it tell Simflex > about it through magic instructions ? If not, what mechanism does simflex use > to report the number of requests completed in the final reported out ? > > 2) I am guessing these workloads setup as a distributed simics simulation, > correct ? For example, for websearch, is the frontend server on a separate > simulated machine, and the driver on a separate simulated machine, and the > index nodes on a separate simulated machine ? If yes, and if the answer to my > question in 1) is yes as well, then for websearch, are the magic instructions > placed in the faban driver to notify whether a request has ended or started ? > > 3) This question is only relevant if simflex does not know when a request was > generated or completed. If the answer to 1) is no, how is performance > measured ? Is the benchmark run for fixed number of instructions in each > sampling phase and then their execution times in each phase is aggregated / > extrapolated to what the most likely execution time would've been if the > entire benchmark were run to completion ? > > Thanks alot >
