Hi Andreas, I agree with this sentiment overall as I think our collection of regressions currently are pretty ad-hoc; they are basically just the benchmarks that people want to ensure run for their in gem5, i.e, SPEC etc. I think smaller LLVM unit tests would provide more directed testing, similar or better coverage, less maintenance, and a suitable license.
Just to clarify when you say 'LLVM-specific' tests, are you referring to the unit/regression tests: http://llvm.org/docs/TestingGuide.html#regression-tests ? We are also thinking of using similar HCC tests (AMD's LLVM-based compiler for heterogeneous accelerators): https://github.com/RadeonOpenCompute/hcc/tree/clang_tot_upgrade/tests/Unit for our GPU model in gem5. For testing, I'm assuming we're going to want something like Jenkins going forward. The question is where will such a server live, and will it be accessible to the public? -Tony -----Original Message----- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas Hansson Sent: Monday, November 14, 2016 2:40 PM To: gem5 Developer List <gem5-dev@gem5.org>; Jason Lowe-Power <ja...@lowepower.com> Subject: Re: [gem5-dev] Status of RISC-V patches Hi all, Merely a thought when it comes to adding tests: I would suggest we ditch all proprietary/closed-source tests and move in the direction of something that is open and maintained. My proposal would be to adopt a few of the tests from the llvm test suite. There are both very short LLVM-specific tests that are BSD licensed, and a bunch of smaller “apps” that maintain their original license. It would be great if we could identify a few relevant tests from the tests suite and start going in that direction. The source for the tests along with build scripts etc should probably be in a repo on their own. What do you think? Andreas On 28/10/2016, 22:12, "gem5-dev on behalf of Alec Roelke" <gem5-dev-boun...@gem5.org on behalf of ar...@virginia.edu> wrote: >I think I get it. Thanks for your help! > >It looks like I won't be able to make use of any tests other than >00.hello, because either they're multithreaded or pieces of them are >missing. I'm going to try to put my instruction tests into >02.insttest. > >On Fri, Oct 28, 2016 at 11:01 AM, Jason Lowe-Power ><ja...@lowepower.com> >wrote: > >> Hi Alec, >> >> Our regression testing infrastructure is confusing, to say the least. >> Honestly, it would take me at least a few hours to figure out how to >>add new regressions again. But here's a little information that >>hopefully will help you save some time. >> >> You need to put reference outputs in tests/quick/se/<TEST >> NAME>/ref/riscv/linux/<CONFIG FILE NAME>. >> >> The config files are in tests/configs. One of these files (without >>the >> .py) is what goes in <CONFIG FILE NAME> above. The <TEST NAME> can be >>anything. Using some of the same tests that are there make sense, but >>you may want to add other RISC-V specific tests as well. >> >> You *may* be able to use the config files in tests/configs without >>changes, but maybe not. I'm not sure. You'll probably be able to use >>some but not others. >> >> Getting scons to run your regressions is totally black magic to me. >>Here's >> a couple of things I've learned, though. >> 1) You should delete the build directory (e.g., build/RISCV/tests/ >> debug/quick/se/00.hello/riscv/) every time you want to re-run the >>regression. >> 2) The regression won't even try to run if you don't have dummy files >>in tests/quick/se/<TEST NAME>/ref/riscv/linux/<CONFIG FILE NAME>. >> >> For the missing stats/changed stats, since this is first time you're >>running the RISC-V stats, I would totally ignore all of that >>information. I would just look at the output (simout/simerr) and make >>sure it seems reasonable (e.g., no errors). If you have an idea of a >>stat that you expect to see (e.g., floating point ops > 0) then you >>could look at the stats file too to make sure it's what you expect. >> >> There's no specific configuration you should be using. For the most >>part, all of the tests are just functional tests. The configurations >>for the system are specified in the tests/configs (<CONFIG FILE >>NAME>) Python files. >> >> Hopefully this will help you get started. Let us know if you have >> more questions. Maybe someone with more regression tester experience >> will be able to help more. >> >> Jason >> >> On Thu, Oct 27, 2016 at 8:23 PM Alec Roelke <ar...@virginia.edu> wrote: >> >>> I'll start with converting as many from "quick" as I can. If/when I >>>end up creating my own, is there a convention for what they should >>>be named and where they should go? >>> >>> Also, I'm having a bit of a problem with just 00.hello. I compiled >>>the source in tests/test-progs/hello/src into >>>tests/test-progs/hello/bin/riscv/linux >>> (for some reason mine disappeared), and then created >>>tests/quick/se/00.hello/ref/riscv/linux/simple-atomic. Then I >>>compiled and ran build/RISCV/gem5.fast, configuring for the atomic >>>CPU and redirecting the output stats, configuration, stdout, and >>>stderr into that directory. But when I run >>>build/RISCV/tests/debug/quick/ >>>se/00.hello/riscv/linux/simple-atomic, it claims that it couldn't >>>find several stats and that it found several other unexpected ones >>>in stats.txt (and completely ignored the other files). Is there >>>some other configuration I should be using to generate the output >>>for the regression? >>> Actually, better yet, is there a way for me to figure out what the >>>configuration I should be using is, since I imagine I'll run into >>>this problem for the other CPU models? >>> >>> On Thu, Oct 27, 2016 at 6:38 PM, Jason Lowe-Power >>> <ja...@lowepower.com> >>> wrote: >>> >>> Hi Alec, >>> >>> Thanks for taking the time for writing tests. It's something that we >>>as a community need to get better at. >>> >>> To respond to your questions: >>> - It is completely acceptable for you to include RISC-V only tests. >>>In fact, I think it's a necessity. >>> - Focusing just on the "quick" regressions makes sense to me. If you >>>have one or two longer benchmarks, that would be good, but not >>>required. If you could stay away from SPEC it would be better. I >>>believe that we can't distribute the binaries, which makes it a pain >>>for others to run the regression test. >>> - Covering corner cases would be amazing, but again, not required. >>>If you look at gem5's current regression suite, you'll find that we >>>currently don't have anything like that. So, if it doesn't take you >>>too much time, this would be a good addition, but just getting >>>coverage of all instructions would be a step up from all the other >>>ISAs. >>> - For m5threads, no need to implement it for RISC-V. Although, it >>>looks like you only need to implement 3 functions, so I don't think >>>it would be too hard. But that's a project for another day :). >>> >>> Again, I want to stress that we can't expect you to spend lots of >>>time writing great regressions. It would be very hypocritical :). >>>Anything is acceptable. Of course, better regressions mean that >>>RISC-V will be more usable and more stable. >>> >>> Cheers, >>> Jason >>> >>> On Thu, Oct 27, 2016 at 5:29 PM Alec Roelke <ar...@virginia.edu> wrote: >>> >>> I'll certainly add regressions for "hello" for each of the four >>>models, and I'll try to get other "quick" tests done the same way, >>>too. I won't be able to do all of them as m5threads hasn't been >>>implemented for RISC-V, but I'll do what I can. I can also do the >>>"long" ones the same way, if time isn't a concern (I noticed some >>>were from SPEC, which could take a long time to complete). >>> >>> Because m5threads hasn't been implemented for RISC-V, and my patches >>>only support SE mode, I can't actually test if the atomic >>>instructions work properly when used concurrently, but I can at >>>least test that they perform the read-modify-write operations >>>properly. Is it okay if I add a few regressions that only work for >>>RISC-V since they'd use assembly calls? >>> For >>> that matter, should I be making sure that the existing regressions >>>cover corner cases in instructions, or is it sufficient to see that >>>each instruction is represented at least once by them? I could >>>write some tests that check corner cases, but at least some would >>>use assembly calls and thus be incompatible with anything other than >>>RISC-V. >>> >>> On Thu, Oct 27, 2016 at 5:55 PM, Jason Lowe-Power >>> <ja...@lowepower.com> >>> wrote: >>> >>> Hi Alec, >>> >>> Thanks again for implementing RISC-V in gem5. It's an incredibly >>> important and timely addition! >>> >>> As far as I can tell, the patches look good. Hopefully some other >>> will review them soon as well. >>> >>> The only thing that's missing that I would really like to have >>>before pushing the patches is some regression tests for RISC-V. If >>>you could look at http://gem5.org/Regression_Tests and have a go at >>>adding some regressions, it would be helpful. It would be *great* if >>>you could make sure the regressions cover most of what you've >>>implemented (e.g., multiply/atomic/etc. instructions, Linux >>>syscalls, etc.). If that isn't possible, at least having a "hello" >>>regression for a couple of different CPU models is needed. >>> >>> Thanks again for your contribution! >>> >>> Jason >>> >>> >>> >>> >_______________________________________________ >gem5-dev mailing list >gem5-dev@gem5.org >http://m5sim.org/mailman/listinfo/gem5-dev IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
