I think I get it. Thanks for your help! It looks like I won't be able to make use of any tests other than 00.hello, because either they're multithreaded or pieces of them are missing. I'm going to try to put my instruction tests into 02.insttest.
On Fri, Oct 28, 2016 at 11:01 AM, Jason Lowe-Power <ja...@lowepower.com> wrote: > Hi Alec, > > Our regression testing infrastructure is confusing, to say the least. > Honestly, it would take me at least a few hours to figure out how to add > new regressions again. But here's a little information that hopefully will > help you save some time. > > You need to put reference outputs in tests/quick/se/<TEST > NAME>/ref/riscv/linux/<CONFIG FILE NAME>. > > The config files are in tests/configs. One of these files (without the > .py) is what goes in <CONFIG FILE NAME> above. The <TEST NAME> can be > anything. Using some of the same tests that are there make sense, but you > may want to add other RISC-V specific tests as well. > > You *may* be able to use the config files in tests/configs without > changes, but maybe not. I'm not sure. You'll probably be able to use some > but not others. > > Getting scons to run your regressions is totally black magic to me. Here's > a couple of things I've learned, though. > 1) You should delete the build directory (e.g., build/RISCV/tests/ > debug/quick/se/00.hello/riscv/) every time you want to re-run the > regression. > 2) The regression won't even try to run if you don't have dummy files in > tests/quick/se/<TEST NAME>/ref/riscv/linux/<CONFIG FILE NAME>. > > For the missing stats/changed stats, since this is first time you're > running the RISC-V stats, I would totally ignore all of that information. I > would just look at the output (simout/simerr) and make sure it seems > reasonable (e.g., no errors). If you have an idea of a stat that you expect > to see (e.g., floating point ops > 0) then you could look at the stats file > too to make sure it's what you expect. > > There's no specific configuration you should be using. For the most part, > all of the tests are just functional tests. The configurations for the > system are specified in the tests/configs (<CONFIG FILE NAME>) Python files. > > Hopefully this will help you get started. Let us know if you have more > questions. Maybe someone with more regression tester experience will be > able to help more. > > Jason > > On Thu, Oct 27, 2016 at 8:23 PM Alec Roelke <ar...@virginia.edu> wrote: > >> I'll start with converting as many from "quick" as I can. If/when I end >> up creating my own, is there a convention for what they should be named and >> where they should go? >> >> Also, I'm having a bit of a problem with just 00.hello. I compiled the >> source in tests/test-progs/hello/src into >> tests/test-progs/hello/bin/riscv/linux >> (for some reason mine disappeared), and then created >> tests/quick/se/00.hello/ref/riscv/linux/simple-atomic. Then I compiled >> and ran build/RISCV/gem5.fast, configuring for the atomic CPU and >> redirecting the output stats, configuration, stdout, and stderr into that >> directory. But when I run build/RISCV/tests/debug/quick/ >> se/00.hello/riscv/linux/simple-atomic, it claims that it couldn't find >> several stats and that it found several other unexpected ones in stats.txt >> (and completely ignored the other files). Is there some other >> configuration I should be using to generate the output for the regression? >> Actually, better yet, is there a way for me to figure out what the >> configuration I should be using is, since I imagine I'll run into this >> problem for the other CPU models? >> >> On Thu, Oct 27, 2016 at 6:38 PM, Jason Lowe-Power <ja...@lowepower.com> >> wrote: >> >> Hi Alec, >> >> Thanks for taking the time for writing tests. It's something that we as a >> community need to get better at. >> >> To respond to your questions: >> - It is completely acceptable for you to include RISC-V only tests. In >> fact, I think it's a necessity. >> - Focusing just on the "quick" regressions makes sense to me. If you have >> one or two longer benchmarks, that would be good, but not required. If you >> could stay away from SPEC it would be better. I believe that we can't >> distribute the binaries, which makes it a pain for others to run the >> regression test. >> - Covering corner cases would be amazing, but again, not required. If you >> look at gem5's current regression suite, you'll find that we currently >> don't have anything like that. So, if it doesn't take you too much time, >> this would be a good addition, but just getting coverage of all >> instructions would be a step up from all the other ISAs. >> - For m5threads, no need to implement it for RISC-V. Although, it looks >> like you only need to implement 3 functions, so I don't think it would be >> too hard. But that's a project for another day :). >> >> Again, I want to stress that we can't expect you to spend lots of time >> writing great regressions. It would be very hypocritical :). Anything is >> acceptable. Of course, better regressions mean that RISC-V will be more >> usable and more stable. >> >> Cheers, >> Jason >> >> On Thu, Oct 27, 2016 at 5:29 PM Alec Roelke <ar...@virginia.edu> wrote: >> >> I'll certainly add regressions for "hello" for each of the four models, >> and I'll try to get other "quick" tests done the same way, too. I won't be >> able to do all of them as m5threads hasn't been implemented for RISC-V, but >> I'll do what I can. I can also do the "long" ones the same way, if time >> isn't a concern (I noticed some were from SPEC, which could take a long >> time to complete). >> >> Because m5threads hasn't been implemented for RISC-V, and my patches only >> support SE mode, I can't actually test if the atomic instructions work >> properly when used concurrently, but I can at least test that they perform >> the read-modify-write operations properly. Is it okay if I add a few >> regressions that only work for RISC-V since they'd use assembly calls? For >> that matter, should I be making sure that the existing regressions cover >> corner cases in instructions, or is it sufficient to see that each >> instruction is represented at least once by them? I could write some tests >> that check corner cases, but at least some would use assembly calls and >> thus be incompatible with anything other than RISC-V. >> >> On Thu, Oct 27, 2016 at 5:55 PM, Jason Lowe-Power <ja...@lowepower.com> >> wrote: >> >> Hi Alec, >> >> Thanks again for implementing RISC-V in gem5. It's an incredibly >> important and timely addition! >> >> As far as I can tell, the patches look good. Hopefully some other will >> review them soon as well. >> >> The only thing that's missing that I would really like to have before >> pushing the patches is some regression tests for RISC-V. If you could look >> at http://gem5.org/Regression_Tests and have a go at adding some >> regressions, it would be helpful. It would be *great* if you could make >> sure the regressions cover most of what you've implemented (e.g., >> multiply/atomic/etc. instructions, Linux syscalls, etc.). If that isn't >> possible, at least having a "hello" regression for a couple of different >> CPU models is needed. >> >> Thanks again for your contribution! >> >> Jason >> >> >> >> _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
