Hi Lluis, It’s not the input/output to/from files that is slow, it is the passing of data to/from C++ and Python. With this framework it would be trivial to add JSON as a new backend, the SQLite is just an example.
An opportunity would be to try and use swig’s —builtin option to try and reduce the overheads. Andreas On 16/04/2014 16:20, "Lluís Vilanova" <[email protected]> wrote: >Andreas Hansson writes: > >> Hi Lluis, >> I don¹t want to hijack the thread, but if you want to use this you need: > >> 1184 - 1186 > >> 1636 - 1647 > >> 1676 - 1678 > >> The performance is indeed closely coupled to how often you dump the >>stats, >> but it is a big enough issue at the moment that it needs to be resolved >> before the patches can go in. > >> SQLite is chosen as it gives a good way to select subsets of results, >>and >> the shelves is a convenient way of storing the data structures such that >> they can be transparently read in again later. > >That's what I was looking for, thanks. > >As a side note, if shelves are only used for temporary storage of python >data >structures, then it could be faster to use cPickle instead. > > >Lluis > > >> On 16/04/2014 15:28, "Lluís Vilanova" <[email protected]> wrote: > >>> I'm curious. Why do you use python shelves instead of plain JSON? Is >>> serialization faster, or just more convenient to implement? >>> >>> As for the performance bottleneck, I imagine it should not be a problem >>> unless >>> you're producing dumps very frequently during the execution. >>> >>> I've also tried to see which patch in the review board adds the python >>> support >>> for statistics; I've found some like http://reviews.gem5.org/r/1636/, >>>but >>> they're all based on something older I could not find. >>> >>> >>> Lluis >>> >>> Andreas Hansson writes: >>> >>>> Hi everyone, >>>> As the issue of having a better stats format popped up, I merely >>>>wanted >>>> to >>>> grab the opportunity and highlight that there are already patches to >>>> turn >>>> the stats into Python data structures, and dump them either as >>>>shelves, >>>> or >>>> in a SQLite database (see e.g. http://reviews.gem5.org/r/1638/ and the >>>> related patches). The only issue is that the performance impact is not >>>> negligible, and the bottleneck seems to be in the C++/Python swig >>>> wrapping. More work is thus needed to improve the performance, but the >>>> functionality is there if needed. >>> >>>> Any volunteers? >>> >>>> Andreas >>> >>> >>>> On 16/04/2014 13:39, "Lluís Vilanova" <[email protected]> wrote: >>> >>>>> Good to know. Then I suppose the correct accounting is: >>>>> >>>>> <stat name="total_cycles" value="{{ core.numCycles }}"/> >>>>> <stat name="idle_cycles" value="{{ core.idleCycles }}"/> >>>>> <stat name="busy_cycles" value="{{ core.numCycles - core.idleCycles >>>>> }}"/> >>>>> >>>>> If so, I'll commit a fix for that. I started writing this code using >>>>>a >>>>> very old >>>>> gem5 version, thus the "oldstable" templates. >>>>> >>>>> BTW, are there any plans for dumping stats in JSON format? That would >>>>> greatly >>>>> simplify post-processing of results, and its very simple to write a >>>>> script that >>>>> dumps JSON stats in the current text format. Another issue I've found >>>>> is >>>>> that >>>>> conditional stats dumping complicates post-processing (e.g., some >>>>>stats >>>>> are not >>>>> dumped if their value is zero). >>>>> >>>>> >>>>> Thanks, >>>>> Lluis >>>>> >>>>> >>>>> Amin Farmahini writes: >>>>> >>>>>> In O3, numCycles includes idleCycles. So busy cycles is numCycles - >>>>>> idleCycles. >>>>>> As a side note, there is also quiesceCycles. >>>>> >>>>>> Thanks, >>>>>> Amin >>>>> >>>>>> On Tue, Apr 15, 2014 at 2:45 PM, Tiago Mück <[email protected]> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Hi Lluíz, >>>>> >>>>> >>>>>> I've noticed that you are using this mapping in your core template: >>>>> >>>>> >>>>> >>>>>> <stat name="total_cycles" value="{{ core.numCycles }}"/> >>>>>> >>>>> >>>>> >>>>>> <stat name="idle_cycles" value="{{ core.iew.iewIdleCycles >>>>>> }}"/> >>>>> >>>>> >>>>>> <stat name="busy_cycles" value="{{ core.numCycles - >>>>>> core.iew.iewIdleCycles }}"/> >>>>> >>>>> >>>>> >>>>>> I think core.numCycles accounts only for busy cycles, so the >>>>>> correct would >>>>>> be: >>>>> >>>>> >>>>> >>>>>> <stat name="total_cycles" value="{{ core.numCycles + >>>>>> core.iew.iewIdleCycles }}"/> >>>>> >>>>> >>>>>> <stat name="idle_cycles" value="{{ core.iew.iewIdleCycles >>>>>> }}"/> >>>>> >>>>> >>>>>> <stat name="busy_cycles" value="{{ core.numCycles}}"/> >>>>>> >>>>> >>>>> >>>>> >>>>>> Please let me know if I'm wrong. >>>>> >>>>> >>>>>> Also, why are you using core.iew.iewIdleCycle instead of >>>>>> core.idleCycles ? >>>>> >>>>> >>>>>> Regards, >>>>> >>>>>> Tiago >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> On Tue, Apr 15, 2014 at 4:13 AM, Lluís Vilanova >>>>>> <[email protected]> wrote: >>>>> >>>>>> Sobhan Niknam writes: >>>>> >>>>>>> Hi, >>>>>>> I need to use GEM5 and McPAT together. Also, I use ALPHA >>>>>>> Architectures >>>>>> over >>>>>>> GEM5. I need to have the power analysis of all components in Gem5 >>>>>> simulation. >>>>>>> for extracting power from output of Gem5 with McPAT, I searched a >>>>>>> lot, >>>>>> and find >>>>>>> valid parsers or methods to integrate GEM5 with McPAT, I tried with >>>>>> the scripts >>>>>>> available, like m5-mcpat.pl, m5-mcpat-parser-se.py but could not >>>>>>>work >>>>>> and I >>>>>>> encountered with errors. >>>>>>> Unfortunately I don't have time for writing parser myself. please, >>>>>> help if you >>>>>>> have working methods/scripts and notes for learning. >>>>>>> also if there is any other alternative for power analysis. >>>>> >>>>>> You could try with this: >>>>> >>>>>> https://projects.gso.ac.upc.edu/projects/gem5-mcpat/repository >>>>> >>>>>> I've recently updated it to parse the results of a newer gem5 >>>>>> version, >>>>>> but you >>>>>> might still have to adapt the templates to your needs (in case >>>>>> you're >>>>>> not using >>>>>> the configuration I needed). In any case, the they are broken >>>>>> down into >>>>>> interchangeable pieces; take a look at the "main" file >>>>>> "gem5_mcpat/templates/example.xml". >>>>> >>>>>> Hope that helps. >>>>> >>>>> >>>>>> Lluis > >-- > "And it's much the same thing with knowledge, for whenever you learn > something new, the whole world becomes that much richer." > -- The Princess of Pure Reason, as told by Norton Juster in The Phantom > Tollbooth >_______________________________________________ >gem5-users mailing list >[email protected] >http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782 _______________________________________________ gem5-users mailing list [email protected] http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
