Re: [m5-dev] changeset in m5: regress: Regression Tester output updates
I'm rolling back this stat update and rerunning/reupdating. It's going to take a while, but I'll push once it's done. Gabe On 02/06/11 23:03, Ali Saidi wrote: This seems like a really half baked attempt to update the stats. You've removed all the ARM FS stats, and you're only seem to have updated a few of the quick tests, however all of the other tests aren't updated. Since you added some stats every test will need an update. Ali On Feb 7, 2011, at 12:17 AM, Brad Beckmann wrote: changeset 05f52a716144 in /z/repo/m5 details: http://repo.m5sim.org/m5?cmd=changeset;node=05f52a716144 description: regress: Regression Tester output updates diffstat: tests/quick/00.hello/ref/alpha/linux/inorder-timing/config.ini |13 +- tests/quick/00.hello/ref/alpha/linux/inorder-timing/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/inorder-timing/stats.txt |10 +- tests/quick/00.hello/ref/alpha/linux/o3-timing/config.ini |13 +- tests/quick/00.hello/ref/alpha/linux/o3-timing/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/o3-timing/stats.txt |31 +- tests/quick/00.hello/ref/alpha/linux/simple-atomic/config.ini |11 +- tests/quick/00.hello/ref/alpha/linux/simple-atomic/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-atomic/stats.txt |24 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MESI_CMP_directory/config.ini |14 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MESI_CMP_directory/ruby.stats |30 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MESI_CMP_directory/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MESI_CMP_directory/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_directory/config.ini |68 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_directory/ruby.stats |54 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_directory/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_directory/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_token/config.ini |68 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_token/ruby.stats |94 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_token/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_CMP_token/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_hammer/config.ini |97 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_hammer/ruby.stats | 164 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_hammer/simout |10 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby-MOESI_hammer/stats.txt |30 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/config.ini | 228 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/ruby.stats | 282 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing/config.ini |13 +- tests/quick/00.hello/ref/alpha/linux/simple-timing/simout |12 +- tests/quick/00.hello/ref/alpha/linux/simple-timing/stats.txt |26 +- tests/quick/00.hello/ref/alpha/tru64/o3-timing/config.ini |13 +- tests/quick/00.hello/ref/alpha/tru64/o3-timing/simout | 8 +- tests/quick/00.hello/ref/alpha/tru64/o3-timing/stats.txt |30 +- tests/quick/00.hello/ref/alpha/tru64/simple-atomic/config.ini |13 +- tests/quick/00.hello/ref/alpha/tru64/simple-atomic/simout |10 +- tests/quick/00.hello/ref/alpha/tru64/simple-atomic/stats.txt |24 +- tests/quick/00.hello/ref/alpha/tru64/simple-timing-ruby-MESI_CMP_directory/config.ini |14 +- tests/quick/00.hello/ref/alpha/tru64/simple-timing-ruby-MESI_CMP_directory/ruby.stats |26 +- tests/quick/00.hello/ref/alpha/tru64/simple-timing-ruby-MESI_CMP_directory/simout | 8 +- tests/quick/00.hello/ref/alpha/tru64/simple-timing-ruby-MESI_CMP_directory/stats.txt |26 +-
Re: [m5-dev] changeset in m5: regress: Regression Tester output updates
Ugh...sorry about that. I had to update most of the stats because one of Joel's patches added several new stats. The problem was that I don't have the Linux kernel to run the ARM FS regression tests. Therefore those tests didn't run correctly and thus I incorrectly updated those regression output files. A similar problem occurred for the X86_SE o3 test. There is no excuse for my incorrect update of these regression output files. However, one thing that will help me in the future is making sure that all of us have the capability to run all regress tests. Many of us, including myself, don't have log in access to zizzer at Michigan, and thus it is very hard for me to reproduce the environment on zizzer, including external file dependencies. Thanks, Brad -Original Message- From: m5-dev-boun...@m5sim.org [mailto:m5-dev-boun...@m5sim.org] On Behalf Of Gabe Black Sent: Monday, February 07, 2011 12:47 AM To: M5 Developer List Subject: Re: [m5-dev] changeset in m5: regress: Regression Tester output updates I'm rolling back this stat update and rerunning/reupdating. It's going to take a while, but I'll push once it's done. Gabe On 02/06/11 23:03, Ali Saidi wrote: This seems like a really half baked attempt to update the stats. You've removed all the ARM FS stats, and you're only seem to have updated a few of the quick tests, however all of the other tests aren't updated. Since you added some stats every test will need an update. Ali On Feb 7, 2011, at 12:17 AM, Brad Beckmann wrote: changeset 05f52a716144 in /z/repo/m5 details: http://repo.m5sim.org/m5?cmd=changeset;node=05f52a716144 description: regress: Regression Tester output updates diffstat: tests/quick/00.hello/ref/alpha/linux/inorder-timing/config.ini |13 +- tests/quick/00.hello/ref/alpha/linux/inorder-timing/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/inorder-timing/stats.txt |10 +- tests/quick/00.hello/ref/alpha/linux/o3-timing/config.ini | 13 +- tests/quick/00.hello/ref/alpha/linux/o3-timing/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/o3-timing/stats.txt | 31 +- tests/quick/00.hello/ref/alpha/linux/simple-atomic/config.ini |11 +- tests/quick/00.hello/ref/alpha/linux/simple-atomic/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-atomic/stats.txt |24 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MESI_CMP_directory/config.ini |14 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MESI_CMP_directory/ruby.stats |30 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MESI_CMP_directory/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MESI_CMP_directory/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_directory/config.ini |68 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_directory/ruby.stats |54 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_directory/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_directory/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_token/config.ini |68 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_token/ruby.stats |94 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_token/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_CMP_token/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_hammer/config.ini|97 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_hammer/ruby.stats| 164 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_hammer/simout|10 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby- MOESI_hammer/stats.txt |30 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/config.ini | 228 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/ruby.stats | 282 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/simout | 8 +- tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby/stats.txt |26 +- tests/quick/00.hello/ref/alpha/linux/simple-timing/config.ini |13 +- tests/quick/00.hello/ref/alpha/linux/simple-timing/simout |12 +- tests/quick/00.hello/ref/alpha/linux/simple-timing/stats.txt |26 +- tests/quick/00.hello/ref/alpha/tru64/o3-timing/config.ini | 13 +- tests/quick/00.hello/ref/alpha/tru64/o3-timing/simout | 8 +- tests/quick/00.hello/ref/alpha/tru64/o3-timing/stats.txt | 30 +- tests/quick/00.hello/ref
Re: [m5-dev] changeset in m5: regress: Regression Tester output updates
On Mon, Feb 7, 2011 at 9:15 AM, Beckmann, Brad brad.beckm...@amd.comwrote: However, one thing that will help me in the future is making sure that all of us have the capability to run all regress tests. Many of us, including myself, don't have log in access to zizzer at Michigan, and thus it is very hard for me to reproduce the environment on zizzer, including external file dependencies. One idea that could be worthwhile would be to set up some kind of regression server on zizzer, that would allow people to run regressions there without necessarily creating logins (sort of the way we do hg access on m5sim.org... in fact it would be ideal to leverage that access control since it's basically the same set of people we want to target). I'm thinking of something where you could push a set of patches to zizzer or daystrom, then remotely kick off a job on zizzer that creates a new repo, pushes those patches, and runs regressions. This would be an interim solution for regressions that use files we can't distribute, and even once that issue is addressed it would still have some value in letting people test with the OS/compiler/tool versions on zizzer so there aren't as many surprises once code is committed and the official regressions run. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] changeset in m5: regress: Regression Tester output updates
Hi Gabe, Yes, the set of patches I checked in require are a lot of changes to the output files. I scanned parts of the regression tester patch last night and noticed those changes as well, including the 5x change (actually it is more like 10x in most cases) in simticks for the mem tester. They all make sense to me. There are multiple patches that impact the regression tester output. The McPAT cpu counter and work unit patches added several new variables to every stats.txt file. That is the major source of changes in those files you listed below (minus the memtester). The large difference in the memtester is something else. Another one of my patches fixed a problem with respect to what block size ruby indicated to the cpus. By fixing this problem, it exposed the fact that ruby did not support the retry semantics expected by the cpu models. I thus added that support, which then fixed a major problem in the memtester. Interestingly enough, indicating a block size of 0 to the memtester caused the memtester to issue only one request at a time per cpu. Now the memtester issues as many requests as possible to ruby until the sequencer's outstanding request count is reached (16 by default). The significantly higher contention is the reason why the memtester simticks increase by 5-10x. Overall, I am aware of the many changes I made to the regression tester output last night. The problem was that my changes were so significant that I failed to realize that I also slipped in completely removing the regression tester output files for the ARM_FS and x86 o3 timing tests. I also noticed last night that I couldn't successfully run the ARM_FS and x86 o3 timing tests locally, but I figured that was ok since those failures were due to environment issues. What I failed to do is put the two things together, and realize that I can't update regression tester output if I can't successfully run those tests. Oh well, live and learn. Thanks Gabe for rerunning those tests and updating the regression tester output! Brad -Original Message- From: m5-dev-boun...@m5sim.org [mailto:m5-dev-boun...@m5sim.org] On Behalf Of Gabe Black Sent: Monday, February 07, 2011 1:59 PM To: M5 Developer List Subject: Re: [m5-dev] changeset in m5: regress: Regression Tester output updates Yeah, unfortunately some of those files we can't distribute, but I'm pretty sure the ARM Linux kernel we can. As we discussed before it would be ideal to move away from the regressions that need files to run that we can't actually give people, but that's likely going to be a lot of work. In any case, the regressions reran and I have an update. I went through all the diffs and saw lots of what I expected (new stats, different host stats, minor config.ini changes, different paths to things) but I also saw a few regressions with unexpected (by me) differences. These were: tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby tests/quick/00.hello/ref/alpha/tru64/simple-timing-ruby tests/quick/00.hello/ref/mips/linux/simple-timing-ruby tests/quick/00.hello/ref/sparc/linux/simple-timing-ruby tests/quick/00.hello/ref/x86/linux/simple-timing-ruby tests/quick/50.memtest/ref/alpha/linux/memtest-ruby tests/quick/60.rubytest/ref/alpha/linux/rubytest-ruby The memtest-ruby regression seems to be the most significantly affected where the number of ticks is increased by a factor of about 5. The patch I made is attached in case anybody wants to go through it. I'd suggest that at least a somebody that's familiar with Ruby go through the tests I pointed out and verify the changes are what they expected. Once one of the Ruby folks (Brad maybe?) lets me know everything is on track and nobody has asked otherwise, I'll go ahead and commit this. Gabe On 02/07/11 09:15, Beckmann, Brad wrote: Ugh...sorry about that. I had to update most of the stats because one of Joel's patches added several new stats. The problem was that I don't have the Linux kernel to run the ARM FS regression tests. Therefore those tests didn't run correctly and thus I incorrectly updated those regression output files. A similar problem occurred for the X86_SE o3 test. There is no excuse for my incorrect update of these regression output files. However, one thing that will help me in the future is making sure that all of us have the capability to run all regress tests. Many of us, including myself, don't have log in access to zizzer at Michigan, and thus it is very hard for me to reproduce the environment on zizzer, including external file dependencies. Thanks, Brad -Original Message- From: m5-dev-boun...@m5sim.org [mailto:m5-dev- boun...@m5sim.org] On Behalf Of Gabe Black Sent: Monday, February 07, 2011 12:47 AM To: M5 Developer List Subject: Re: [m5-dev] changeset in m5: regress: Regression Tester output updates I'm rolling back this stat update and rerunning/reupdating
Re: [m5-dev] changeset in m5: regress: Regression Tester output updates
No problem. I just wanted to make sure we weren't papering over any problems by accident. The O3 tests are probably failing because it's not picking up the fact that it should build the O3 cpu. You can either tell it to from the scons command line, or delete the entire build directory (not just X86_*) to get it to pull in the defaults. Gabe On 02/07/11 16:53, Beckmann, Brad wrote: Hi Gabe, Yes, the set of patches I checked in require are a lot of changes to the output files. I scanned parts of the regression tester patch last night and noticed those changes as well, including the 5x change (actually it is more like 10x in most cases) in simticks for the mem tester. They all make sense to me. There are multiple patches that impact the regression tester output. The McPAT cpu counter and work unit patches added several new variables to every stats.txt file. That is the major source of changes in those files you listed below (minus the memtester). The large difference in the memtester is something else. Another one of my patches fixed a problem with respect to what block size ruby indicated to the cpus. By fixing this problem, it exposed the fact that ruby did not support the retry semantics expected by the cpu models. I thus added that support, which then fixed a major problem in the memtester. Interestingly enough, indicating a block size of 0 to the memtester caused the memtester to issue only one request at a time per cpu. Now the memtester issues as many requests as possible to ruby until the sequencer's outstanding request count is reached (16 by default). The significantly higher contention is the reason why the memtester simticks increase by 5-10x. Overall, I am aware of the many changes I made to the regression tester output last night. The problem was that my changes were so significant that I failed to realize that I also slipped in completely removing the regression tester output files for the ARM_FS and x86 o3 timing tests. I also noticed last night that I couldn't successfully run the ARM_FS and x86 o3 timing tests locally, but I figured that was ok since those failures were due to environment issues. What I failed to do is put the two things together, and realize that I can't update regression tester output if I can't successfully run those tests. Oh well, live and learn. Thanks Gabe for rerunning those tests and updating the regression tester output! Brad -Original Message- From: m5-dev-boun...@m5sim.org [mailto:m5-dev-boun...@m5sim.org] On Behalf Of Gabe Black Sent: Monday, February 07, 2011 1:59 PM To: M5 Developer List Subject: Re: [m5-dev] changeset in m5: regress: Regression Tester output updates Yeah, unfortunately some of those files we can't distribute, but I'm pretty sure the ARM Linux kernel we can. As we discussed before it would be ideal to move away from the regressions that need files to run that we can't actually give people, but that's likely going to be a lot of work. In any case, the regressions reran and I have an update. I went through all the diffs and saw lots of what I expected (new stats, different host stats, minor config.ini changes, different paths to things) but I also saw a few regressions with unexpected (by me) differences. These were: tests/quick/00.hello/ref/alpha/linux/simple-timing-ruby tests/quick/00.hello/ref/alpha/tru64/simple-timing-ruby tests/quick/00.hello/ref/mips/linux/simple-timing-ruby tests/quick/00.hello/ref/sparc/linux/simple-timing-ruby tests/quick/00.hello/ref/x86/linux/simple-timing-ruby tests/quick/50.memtest/ref/alpha/linux/memtest-ruby tests/quick/60.rubytest/ref/alpha/linux/rubytest-ruby The memtest-ruby regression seems to be the most significantly affected where the number of ticks is increased by a factor of about 5. The patch I made is attached in case anybody wants to go through it. I'd suggest that at least a somebody that's familiar with Ruby go through the tests I pointed out and verify the changes are what they expected. Once one of the Ruby folks (Brad maybe?) lets me know everything is on track and nobody has asked otherwise, I'll go ahead and commit this. Gabe On 02/07/11 09:15, Beckmann, Brad wrote: Ugh...sorry about that. I had to update most of the stats because one of Joel's patches added several new stats. The problem was that I don't have the Linux kernel to run the ARM FS regression tests. Therefore those tests didn't run correctly and thus I incorrectly updated those regression output files. A similar problem occurred for the X86_SE o3 test. There is no excuse for my incorrect update of these regression output files. However, one thing that will help me in the future is making sure that all of us have the capability to run all regress tests. Many of us, including myself, don't have log in access to zizzer at Michigan, and thus it is very hard for me to reproduce the environment
