Hi Andreas, I don't think it's surprising, but I like this idea. I think it's much more maintainable going forward than what we've been doing.
Does this have any bearing on the RISC-V patches? I was hoping we would be able to get Alec's stuff pushed in soon. I don't want to hold things up for perfection, especially since it seems a number of people have been asking of RISC-V support. Thanks, Jason On Mon, Nov 14, 2016 at 4:40 PM Andreas Hansson <andreas.hans...@arm.com> wrote: > Hi all, > > Merely a thought when it comes to adding tests: > > I would suggest we ditch all proprietary/closed-source tests and move in > the direction of something that is open and maintained. My proposal would > be to adopt a few of the tests from the llvm test suite. There are both > very short LLVM-specific tests that are BSD licensed, and a bunch of > smaller “apps” that maintain their original license. It would be great if > we could identify a few relevant tests from the tests suite and start > going in that direction. The source for the tests along with build scripts > etc should probably be in a repo on their own. > > What do you think? > > Andreas > > On 28/10/2016, 22:12, "gem5-dev on behalf of Alec Roelke" > <gem5-dev-boun...@gem5.org on behalf of ar...@virginia.edu> wrote: > > >I think I get it. Thanks for your help! > > > >It looks like I won't be able to make use of any tests other than > >00.hello, > >because either they're multithreaded or pieces of them are missing. I'm > >going to try to put my instruction tests into 02.insttest. > > > >On Fri, Oct 28, 2016 at 11:01 AM, Jason Lowe-Power <ja...@lowepower.com> > >wrote: > > > >> Hi Alec, > >> > >> Our regression testing infrastructure is confusing, to say the least. > >> Honestly, it would take me at least a few hours to figure out how to add > >> new regressions again. But here's a little information that hopefully > >>will > >> help you save some time. > >> > >> You need to put reference outputs in tests/quick/se/<TEST > >> NAME>/ref/riscv/linux/<CONFIG FILE NAME>. > >> > >> The config files are in tests/configs. One of these files (without the > >> .py) is what goes in <CONFIG FILE NAME> above. The <TEST NAME> can be > >> anything. Using some of the same tests that are there make sense, but > >>you > >> may want to add other RISC-V specific tests as well. > >> > >> You *may* be able to use the config files in tests/configs without > >> changes, but maybe not. I'm not sure. You'll probably be able to use > >>some > >> but not others. > >> > >> Getting scons to run your regressions is totally black magic to me. > >>Here's > >> a couple of things I've learned, though. > >> 1) You should delete the build directory (e.g., build/RISCV/tests/ > >> debug/quick/se/00.hello/riscv/) every time you want to re-run the > >> regression. > >> 2) The regression won't even try to run if you don't have dummy files in > >> tests/quick/se/<TEST NAME>/ref/riscv/linux/<CONFIG FILE NAME>. > >> > >> For the missing stats/changed stats, since this is first time you're > >> running the RISC-V stats, I would totally ignore all of that > >>information. I > >> would just look at the output (simout/simerr) and make sure it seems > >> reasonable (e.g., no errors). If you have an idea of a stat that you > >>expect > >> to see (e.g., floating point ops > 0) then you could look at the stats > >>file > >> too to make sure it's what you expect. > >> > >> There's no specific configuration you should be using. For the most > >>part, > >> all of the tests are just functional tests. The configurations for the > >> system are specified in the tests/configs (<CONFIG FILE NAME>) Python > >>files. > >> > >> Hopefully this will help you get started. Let us know if you have more > >> questions. Maybe someone with more regression tester experience will be > >> able to help more. > >> > >> Jason > >> > >> On Thu, Oct 27, 2016 at 8:23 PM Alec Roelke <ar...@virginia.edu> wrote: > >> > >>> I'll start with converting as many from "quick" as I can. If/when I > >>>end > >>> up creating my own, is there a convention for what they should be > >>>named and > >>> where they should go? > >>> > >>> Also, I'm having a bit of a problem with just 00.hello. I compiled the > >>> source in tests/test-progs/hello/src into > >>>tests/test-progs/hello/bin/riscv/linux > >>> (for some reason mine disappeared), and then created > >>> tests/quick/se/00.hello/ref/riscv/linux/simple-atomic. Then I compiled > >>> and ran build/RISCV/gem5.fast, configuring for the atomic CPU and > >>> redirecting the output stats, configuration, stdout, and stderr into > >>>that > >>> directory. But when I run build/RISCV/tests/debug/quick/ > >>> se/00.hello/riscv/linux/simple-atomic, it claims that it couldn't find > >>> several stats and that it found several other unexpected ones in > >>>stats.txt > >>> (and completely ignored the other files). Is there some other > >>> configuration I should be using to generate the output for the > >>>regression? > >>> Actually, better yet, is there a way for me to figure out what the > >>> configuration I should be using is, since I imagine I'll run into this > >>> problem for the other CPU models? > >>> > >>> On Thu, Oct 27, 2016 at 6:38 PM, Jason Lowe-Power <ja...@lowepower.com > > > >>> wrote: > >>> > >>> Hi Alec, > >>> > >>> Thanks for taking the time for writing tests. It's something that we > >>>as a > >>> community need to get better at. > >>> > >>> To respond to your questions: > >>> - It is completely acceptable for you to include RISC-V only tests. In > >>> fact, I think it's a necessity. > >>> - Focusing just on the "quick" regressions makes sense to me. If you > >>>have > >>> one or two longer benchmarks, that would be good, but not required. If > >>>you > >>> could stay away from SPEC it would be better. I believe that we can't > >>> distribute the binaries, which makes it a pain for others to run the > >>> regression test. > >>> - Covering corner cases would be amazing, but again, not required. If > >>>you > >>> look at gem5's current regression suite, you'll find that we currently > >>> don't have anything like that. So, if it doesn't take you too much > >>>time, > >>> this would be a good addition, but just getting coverage of all > >>> instructions would be a step up from all the other ISAs. > >>> - For m5threads, no need to implement it for RISC-V. Although, it looks > >>> like you only need to implement 3 functions, so I don't think it would > >>>be > >>> too hard. But that's a project for another day :). > >>> > >>> Again, I want to stress that we can't expect you to spend lots of time > >>> writing great regressions. It would be very hypocritical :). Anything > >>>is > >>> acceptable. Of course, better regressions mean that RISC-V will be more > >>> usable and more stable. > >>> > >>> Cheers, > >>> Jason > >>> > >>> On Thu, Oct 27, 2016 at 5:29 PM Alec Roelke <ar...@virginia.edu> > wrote: > >>> > >>> I'll certainly add regressions for "hello" for each of the four models, > >>> and I'll try to get other "quick" tests done the same way, too. I > >>>won't be > >>> able to do all of them as m5threads hasn't been implemented for > >>>RISC-V, but > >>> I'll do what I can. I can also do the "long" ones the same way, if > >>>time > >>> isn't a concern (I noticed some were from SPEC, which could take a long > >>> time to complete). > >>> > >>> Because m5threads hasn't been implemented for RISC-V, and my patches > >>>only > >>> support SE mode, I can't actually test if the atomic instructions work > >>> properly when used concurrently, but I can at least test that they > >>>perform > >>> the read-modify-write operations properly. Is it okay if I add a few > >>> regressions that only work for RISC-V since they'd use assembly calls? > >>> For > >>> that matter, should I be making sure that the existing regressions > >>>cover > >>> corner cases in instructions, or is it sufficient to see that each > >>> instruction is represented at least once by them? I could write some > >>>tests > >>> that check corner cases, but at least some would use assembly calls and > >>> thus be incompatible with anything other than RISC-V. > >>> > >>> On Thu, Oct 27, 2016 at 5:55 PM, Jason Lowe-Power <ja...@lowepower.com > > > >>> wrote: > >>> > >>> Hi Alec, > >>> > >>> Thanks again for implementing RISC-V in gem5. It's an incredibly > >>> important and timely addition! > >>> > >>> As far as I can tell, the patches look good. Hopefully some other will > >>> review them soon as well. > >>> > >>> The only thing that's missing that I would really like to have before > >>> pushing the patches is some regression tests for RISC-V. If you could > >>>look > >>> at http://gem5.org/Regression_Tests and have a go at adding some > >>> regressions, it would be helpful. It would be *great* if you could make > >>> sure the regressions cover most of what you've implemented (e.g., > >>> multiply/atomic/etc. instructions, Linux syscalls, etc.). If that isn't > >>> possible, at least having a "hello" regression for a couple of > >>>different > >>> CPU models is needed. > >>> > >>> Thanks again for your contribution! > >>> > >>> Jason > >>> > >>> > >>> > >>> > >_______________________________________________ > >gem5-dev mailing list > >gem5-dev@gem5.org > >http://m5sim.org/mailman/listinfo/gem5-dev > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
