Re: [gem5-dev] Continuous integration is live!
t;> means it won't take a lot of time/computation >>> 2) Long regressions (ruby protocols are tested here) are run every night >>> on a batch of MERGED commits. >>> Which means at a certain point we just checkout origin/HEAD and we run >>> long regressions. >>> >>> This is the setup I would actually recommend: a sanity suite (quick) >>> being run on a commit base, and more serious tests >>> (long) being run periodically. If you are scared about overloading the >>> framework, you can always scale down our ambition >>> and run long regressions every two nights. >>> Running less frequently is better than not running at all >>> >>> Please let me know what you think about this; other devs are welcome to >>> comment as well, >>> >>> Giacomo >>> >>> >>> -- >>> *From:* Jason Lowe-Power >>> *Sent:* 02 May 2019 22:59 >>> *To:* Giacomo Travaglini; Rahul Thakur >>> *Cc:* gem5 Developer List >>> *Subject:* Re: [gem5-dev] Continuous integration is live! >>> >>> Hi Giacomo, >>> >>> In tests/main.py we call scons and use the current environment defaults >>> to build gem5. I don't know if the kokoro infrastructure supports other >>> compilers. This might be something that Rahul can address. >>> >>> I'm also not sure if we can find a way to run more compilations in >>> parallel on Kokoro. I'm happy to refactor the test scripts to do this. >>> However, as it is, we are currently compiling at least 4 binaries mostly >>> sequentially, which is making the testing take a significant amount of >>> time. If we add more compilers (and more Ruby protocols), this is going to >>> begin to get out of hand. It would also be good to compile .fast, .opt, and >>> .debug, but I believe we're only compiling .opt right now. >>> >>> Cheers, >>> Jason >>> >>> On Thu, May 2, 2019 at 5:53 AM Giacomo Travaglini < >>> giacomo.travagl...@arm.com> wrote: >>> >>> Hi Jason, >>> >>> I understand; Another thing I would like to ask: >>> >>> Which script is building gem5 in jenkins? Ideally it would be nice to >>> build with BOTH gcc and clang (so that we avoid >>> periodic "fix clang build" patches. I would also make the version >>> configurable/visible from the script so that >>> we can track changes in compiler support and people can compare failures >>> in case they managed to build >>> seamlessly on their local workspace >>> >>> Giacomo >>> -- >>> *From:* Jason Lowe-Power >>> *Sent:* 26 April 2019 17:49 >>> *To:* Giacomo Travaglini >>> *Cc:* gem5 Developer List >>> *Subject:* Re: [gem5-dev] Continuous integration is live! >>> >>> Hi Giacomo, >>> >>> You *do* have permission :). Anyone can modify >>> tests/jenkins/presubmit.cfg and presubmit.sh. In fact, if you look at the >>> history of the presubmit.sh, it *was* running the old tests. See >>> https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, >>> for instance. >>> >>> The problem is that we can't distribute most of the binaries (e.g., SPEC >>> binaries). We could probably upload them to a private location on the >>> Google Cloud and have jenkins consume them that way, but I believe that >>> will be more work than it's worth. >>> >>> I personally believe that putting effort into porting tests is more >>> worth everyone's time than trying to get the old tests to run, but that's >>> just my opinion. I'm happy to merge changes to run the old tests. I >>> personally believe we should only merge tests into the verification tester >>> which everyone can run locally, but I'm open to proprietary tests, >>> especially in the short term if we have a plan to make them not proprietary. >>> >>> Cheers, >>> Jason >>> >>> On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini < >>> giacomo.travagl...@arm.com> wrote: >>> >>> Hi Jason, >>> >>> It's really amazing that we have a testing framework in place, thanks >>> for your effort! >>> At the moment as far as I can tell we are only running tests registered >>> within the new >>> testing library. >>> >>> I was wondering if we could temporarily enable the system to run legacy >>>
Re: [gem5-dev] Continuous integration is live!
Thanks for the info, Rahul! Is there any way to specify a different compiler (e.g., clang) when building on kokoro? Thanks, Jason On Fri, May 3, 2019 at 9:12 AM Rahul Thakur wrote: > Hi Giacomo, Jason, > > Sorry for late reply. > Re: tool chain - scons and python are available in the test environment > AFAIK. > Re: test throughput - How long does kokoro take to finish current > presubmit test? > > If you have much longer tests to run at periodic frequency, say once in a > few days, on HEAD, kokoro also has a periodic continuous build mode. Such > test will send out email report, serving as a pulse check for ToT. > Presubmit jobs will not be influenced. I can look in to periodic builds if > it fits for gem5 long regression use case. > > RT > > On Fri, May 3, 2019 at 6:17 AM Giacomo Travaglini < > giacomo.travagl...@arm.com> wrote: > >> Hi Jason, >> >> I have seen patches being under review for a long time, and IMHO adding >> an extra 10-30 mins is not the real bottleneck. >> I'd rather wait a little bit more but being sure I am not breaking >> anything... >> >> about ruby protocols, let me say that we (in arm) are relatively happy >> with the current setup: >> >> 1) Quick regressions are run on a commit base (your presubmit.sh). This >> means it won't take a lot of time/computation >> 2) Long regressions (ruby protocols are tested here) are run every night >> on a batch of MERGED commits. >> Which means at a certain point we just checkout origin/HEAD and we run >> long regressions. >> >> This is the setup I would actually recommend: a sanity suite (quick) >> being run on a commit base, and more serious tests >> (long) being run periodically. If you are scared about overloading the >> framework, you can always scale down our ambition >> and run long regressions every two nights. >> Running less frequently is better than not running at all >> >> Please let me know what you think about this; other devs are welcome to >> comment as well, >> >> Giacomo >> >> >> -- >> *From:* Jason Lowe-Power >> *Sent:* 02 May 2019 22:59 >> *To:* Giacomo Travaglini; Rahul Thakur >> *Cc:* gem5 Developer List >> *Subject:* Re: [gem5-dev] Continuous integration is live! >> >> Hi Giacomo, >> >> In tests/main.py we call scons and use the current environment defaults >> to build gem5. I don't know if the kokoro infrastructure supports other >> compilers. This might be something that Rahul can address. >> >> I'm also not sure if we can find a way to run more compilations in >> parallel on Kokoro. I'm happy to refactor the test scripts to do this. >> However, as it is, we are currently compiling at least 4 binaries mostly >> sequentially, which is making the testing take a significant amount of >> time. If we add more compilers (and more Ruby protocols), this is going to >> begin to get out of hand. It would also be good to compile .fast, .opt, and >> .debug, but I believe we're only compiling .opt right now. >> >> Cheers, >> Jason >> >> On Thu, May 2, 2019 at 5:53 AM Giacomo Travaglini < >> giacomo.travagl...@arm.com> wrote: >> >> Hi Jason, >> >> I understand; Another thing I would like to ask: >> >> Which script is building gem5 in jenkins? Ideally it would be nice to >> build with BOTH gcc and clang (so that we avoid >> periodic "fix clang build" patches. I would also make the version >> configurable/visible from the script so that >> we can track changes in compiler support and people can compare failures >> in case they managed to build >> seamlessly on their local workspace >> >> Giacomo >> -- >> *From:* Jason Lowe-Power >> *Sent:* 26 April 2019 17:49 >> *To:* Giacomo Travaglini >> *Cc:* gem5 Developer List >> *Subject:* Re: [gem5-dev] Continuous integration is live! >> >> Hi Giacomo, >> >> You *do* have permission :). Anyone can modify >> tests/jenkins/presubmit.cfg and presubmit.sh. In fact, if you look at the >> history of the presubmit.sh, it *was* running the old tests. See >> https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, >> for instance. >> >> The problem is that we can't distribute most of the binaries (e.g., SPEC >> binaries). We could probably upload them to a private location on the >> Google Cloud and have jenkins consume them that way, but I believe that >> will be more work than it's worth. >> >> I personally
Re: [gem5-dev] Continuous integration is live!
Hi Giacomo, Jason, Sorry for late reply. Re: tool chain - scons and python are available in the test environment AFAIK. Re: test throughput - How long does kokoro take to finish current presubmit test? If you have much longer tests to run at periodic frequency, say once in a few days, on HEAD, kokoro also has a periodic continuous build mode. Such test will send out email report, serving as a pulse check for ToT. Presubmit jobs will not be influenced. I can look in to periodic builds if it fits for gem5 long regression use case. RT On Fri, May 3, 2019 at 6:17 AM Giacomo Travaglini < giacomo.travagl...@arm.com> wrote: > Hi Jason, > > I have seen patches being under review for a long time, and IMHO adding an > extra 10-30 mins is not the real bottleneck. > I'd rather wait a little bit more but being sure I am not breaking > anything... > > about ruby protocols, let me say that we (in arm) are relatively happy > with the current setup: > > 1) Quick regressions are run on a commit base (your presubmit.sh). This > means it won't take a lot of time/computation > 2) Long regressions (ruby protocols are tested here) are run every night > on a batch of MERGED commits. > Which means at a certain point we just checkout origin/HEAD and we run > long regressions. > > This is the setup I would actually recommend: a sanity suite (quick) being > run on a commit base, and more serious tests > (long) being run periodically. If you are scared about overloading the > framework, you can always scale down our ambition > and run long regressions every two nights. > Running less frequently is better than not running at all > > Please let me know what you think about this; other devs are welcome to > comment as well, > > Giacomo > > > -- > *From:* Jason Lowe-Power > *Sent:* 02 May 2019 22:59 > *To:* Giacomo Travaglini; Rahul Thakur > *Cc:* gem5 Developer List > *Subject:* Re: [gem5-dev] Continuous integration is live! > > Hi Giacomo, > > In tests/main.py we call scons and use the current environment defaults to > build gem5. I don't know if the kokoro infrastructure supports other > compilers. This might be something that Rahul can address. > > I'm also not sure if we can find a way to run more compilations in > parallel on Kokoro. I'm happy to refactor the test scripts to do this. > However, as it is, we are currently compiling at least 4 binaries mostly > sequentially, which is making the testing take a significant amount of > time. If we add more compilers (and more Ruby protocols), this is going to > begin to get out of hand. It would also be good to compile .fast, .opt, and > .debug, but I believe we're only compiling .opt right now. > > Cheers, > Jason > > On Thu, May 2, 2019 at 5:53 AM Giacomo Travaglini < > giacomo.travagl...@arm.com> wrote: > > Hi Jason, > > I understand; Another thing I would like to ask: > > Which script is building gem5 in jenkins? Ideally it would be nice to > build with BOTH gcc and clang (so that we avoid > periodic "fix clang build" patches. I would also make the version > configurable/visible from the script so that > we can track changes in compiler support and people can compare failures > in case they managed to build > seamlessly on their local workspace > > Giacomo > -- > *From:* Jason Lowe-Power > *Sent:* 26 April 2019 17:49 > *To:* Giacomo Travaglini > *Cc:* gem5 Developer List > *Subject:* Re: [gem5-dev] Continuous integration is live! > > Hi Giacomo, > > You *do* have permission :). Anyone can modify tests/jenkins/presubmit.cfg > and presubmit.sh. In fact, if you look at the history of the presubmit.sh, > it *was* running the old tests. See > https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, > for instance. > > The problem is that we can't distribute most of the binaries (e.g., SPEC > binaries). We could probably upload them to a private location on the > Google Cloud and have jenkins consume them that way, but I believe that > will be more work than it's worth. > > I personally believe that putting effort into porting tests is more worth > everyone's time than trying to get the old tests to run, but that's just > my opinion. I'm happy to merge changes to run the old tests. I personally > believe we should only merge tests into the verification tester which > everyone can run locally, but I'm open to proprietary tests, especially in > the short term if we have a plan to make them not proprietary. > > Cheers, > Jason > > On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini < > giacomo.travagl...@arm.com> wrote: > > Hi Jason, > > It's really amazing
Re: [gem5-dev] Continuous integration is live!
Hi Jason, I have seen patches being under review for a long time, and IMHO adding an extra 10-30 mins is not the real bottleneck. I'd rather wait a little bit more but being sure I am not breaking anything... about ruby protocols, let me say that we (in arm) are relatively happy with the current setup: 1) Quick regressions are run on a commit base (your presubmit.sh). This means it won't take a lot of time/computation 2) Long regressions (ruby protocols are tested here) are run every night on a batch of MERGED commits. Which means at a certain point we just checkout origin/HEAD and we run long regressions. This is the setup I would actually recommend: a sanity suite (quick) being run on a commit base, and more serious tests (long) being run periodically. If you are scared about overloading the framework, you can always scale down our ambition and run long regressions every two nights. Running less frequently is better than not running at all Please let me know what you think about this; other devs are welcome to comment as well, Giacomo From: Jason Lowe-Power Sent: 02 May 2019 22:59 To: Giacomo Travaglini; Rahul Thakur Cc: gem5 Developer List Subject: Re: [gem5-dev] Continuous integration is live! Hi Giacomo, In tests/main.py we call scons and use the current environment defaults to build gem5. I don't know if the kokoro infrastructure supports other compilers. This might be something that Rahul can address. I'm also not sure if we can find a way to run more compilations in parallel on Kokoro. I'm happy to refactor the test scripts to do this. However, as it is, we are currently compiling at least 4 binaries mostly sequentially, which is making the testing take a significant amount of time. If we add more compilers (and more Ruby protocols), this is going to begin to get out of hand. It would also be good to compile .fast, .opt, and .debug, but I believe we're only compiling .opt right now. Cheers, Jason On Thu, May 2, 2019 at 5:53 AM Giacomo Travaglini mailto:giacomo.travagl...@arm.com>> wrote: Hi Jason, I understand; Another thing I would like to ask: Which script is building gem5 in jenkins? Ideally it would be nice to build with BOTH gcc and clang (so that we avoid periodic "fix clang build" patches. I would also make the version configurable/visible from the script so that we can track changes in compiler support and people can compare failures in case they managed to build seamlessly on their local workspace Giacomo From: Jason Lowe-Power mailto:ja...@lowepower.com>> Sent: 26 April 2019 17:49 To: Giacomo Travaglini Cc: gem5 Developer List Subject: Re: [gem5-dev] Continuous integration is live! Hi Giacomo, You *do* have permission :). Anyone can modify tests/jenkins/presubmit.cfg and presubmit.sh. In fact, if you look at the history of the presubmit.sh, it *was* running the old tests. See https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, for instance. The problem is that we can't distribute most of the binaries (e.g., SPEC binaries). We could probably upload them to a private location on the Google Cloud and have jenkins consume them that way, but I believe that will be more work than it's worth. I personally believe that putting effort into porting tests is more worth everyone's time than trying to get the old tests to run, but that's just my opinion. I'm happy to merge changes to run the old tests. I personally believe we should only merge tests into the verification tester which everyone can run locally, but I'm open to proprietary tests, especially in the short term if we have a plan to make them not proprietary. Cheers, Jason On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini mailto:giacomo.travagl...@arm.com>> wrote: Hi Jason, It's really amazing that we have a testing framework in place, thanks for your effort! At the moment as far as I can tell we are only running tests registered within the new testing library. I was wondering if we could temporarily enable the system to run legacy quick regressions as well, while waiting for porting those to the new library. I guess it is something that shouldn't require a lot of work (just calling .util/regress I guess) I am saying this since a patch recently merged broke some syscall emulation tests and I think it would be beneficial for us to run the entire test suite straightaway while porting tests manually. I could even handle it myself if I had permission to configure the system. Let me know your thoughts, Giacomo From: gem5-dev mailto:gem5-dev-boun...@gem5.org>> on behalf of Jason Lowe-Power mailto:ja...@lowepower.com>> Sent: 16 April 2019 16:30 To: gem5 Developer List; Rahul Thakur Subject: [gem5-dev] Continuous integration is live! Hi all, We now have initial support for continuous integration testing! We should al
Re: [gem5-dev] Continuous integration is live!
Hi Giacomo, In tests/main.py we call scons and use the current environment defaults to build gem5. I don't know if the kokoro infrastructure supports other compilers. This might be something that Rahul can address. I'm also not sure if we can find a way to run more compilations in parallel on Kokoro. I'm happy to refactor the test scripts to do this. However, as it is, we are currently compiling at least 4 binaries mostly sequentially, which is making the testing take a significant amount of time. If we add more compilers (and more Ruby protocols), this is going to begin to get out of hand. It would also be good to compile .fast, .opt, and .debug, but I believe we're only compiling .opt right now. Cheers, Jason On Thu, May 2, 2019 at 5:53 AM Giacomo Travaglini < giacomo.travagl...@arm.com> wrote: > Hi Jason, > > I understand; Another thing I would like to ask: > > Which script is building gem5 in jenkins? Ideally it would be nice to > build with BOTH gcc and clang (so that we avoid > periodic "fix clang build" patches. I would also make the version > configurable/visible from the script so that > we can track changes in compiler support and people can compare failures > in case they managed to build > seamlessly on their local workspace > > Giacomo > -- > *From:* Jason Lowe-Power > *Sent:* 26 April 2019 17:49 > *To:* Giacomo Travaglini > *Cc:* gem5 Developer List > *Subject:* Re: [gem5-dev] Continuous integration is live! > > Hi Giacomo, > > You *do* have permission :). Anyone can modify tests/jenkins/presubmit.cfg > and presubmit.sh. In fact, if you look at the history of the presubmit.sh, > it *was* running the old tests. See > https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, > for instance. > > The problem is that we can't distribute most of the binaries (e.g., SPEC > binaries). We could probably upload them to a private location on the > Google Cloud and have jenkins consume them that way, but I believe that > will be more work than it's worth. > > I personally believe that putting effort into porting tests is more worth > everyone's time than trying to get the old tests to run, but that's just > my opinion. I'm happy to merge changes to run the old tests. I personally > believe we should only merge tests into the verification tester which > everyone can run locally, but I'm open to proprietary tests, especially in > the short term if we have a plan to make them not proprietary. > > Cheers, > Jason > > On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini < > giacomo.travagl...@arm.com> wrote: > > Hi Jason, > > It's really amazing that we have a testing framework in place, thanks for > your effort! > At the moment as far as I can tell we are only running tests registered > within the new > testing library. > > I was wondering if we could temporarily enable the system to run legacy > quick regressions as well, > while waiting for porting those to the new library. I guess it is > something that shouldn't require a lot of work > > (just calling .util/regress I guess) > > I am saying this since a patch recently merged broke some syscall > emulation tests and I think it would > be beneficial for us to run the entire test suite straightaway while > porting tests manually. > I could even handle it myself if I had permission to configure the system. > > Let me know your thoughts, > > Giacomo > > -- > *From:* gem5-dev on behalf of Jason > Lowe-Power > *Sent:* 16 April 2019 16:30 > *To:* gem5 Developer List; Rahul Thakur > *Subject:* [gem5-dev] Continuous integration is live! > > Hi all, > > We now have initial support for continuous integration testing! We should > all thank Google for donating the CPU time and infrastructure to run these > tests. Specifically, Rahul Thakur has been incredibly helpful for the past > two years in getting this off the ground. Thanks, Rahul and the rest of the > team at Google who has been helping us set this up! > > Now, if you submit a patch to gerrit and receive a maintainer +1, "kokoro" > will kick off a build / test of gem5. Once that is complete, you will > receive a verified +1. If it fails, you will receive a verified -1. The > logs can be viewed by anyone once the job is completed by following the > link posted by kokoro (the https://source.cloud.google.com, not the sponge > link). You can see an example on a patch I recently submitted here: > https://gem5-review.googlesource.com/c/public/gem5/+/18068. Note that the > tests take a couple of hours to run. However, I believe there is no limit > to the number of different changes that can be tested at the same time. > &
Re: [gem5-dev] Continuous integration is live!
Hi Jason, I understand; Another thing I would like to ask: Which script is building gem5 in jenkins? Ideally it would be nice to build with BOTH gcc and clang (so that we avoid periodic "fix clang build" patches. I would also make the version configurable/visible from the script so that we can track changes in compiler support and people can compare failures in case they managed to build seamlessly on their local workspace Giacomo From: Jason Lowe-Power Sent: 26 April 2019 17:49 To: Giacomo Travaglini Cc: gem5 Developer List Subject: Re: [gem5-dev] Continuous integration is live! Hi Giacomo, You *do* have permission :). Anyone can modify tests/jenkins/presubmit.cfg and presubmit.sh. In fact, if you look at the history of the presubmit.sh, it *was* running the old tests. See https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, for instance. The problem is that we can't distribute most of the binaries (e.g., SPEC binaries). We could probably upload them to a private location on the Google Cloud and have jenkins consume them that way, but I believe that will be more work than it's worth. I personally believe that putting effort into porting tests is more worth everyone's time than trying to get the old tests to run, but that's just my opinion. I'm happy to merge changes to run the old tests. I personally believe we should only merge tests into the verification tester which everyone can run locally, but I'm open to proprietary tests, especially in the short term if we have a plan to make them not proprietary. Cheers, Jason On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini mailto:giacomo.travagl...@arm.com>> wrote: Hi Jason, It's really amazing that we have a testing framework in place, thanks for your effort! At the moment as far as I can tell we are only running tests registered within the new testing library. I was wondering if we could temporarily enable the system to run legacy quick regressions as well, while waiting for porting those to the new library. I guess it is something that shouldn't require a lot of work (just calling .util/regress I guess) I am saying this since a patch recently merged broke some syscall emulation tests and I think it would be beneficial for us to run the entire test suite straightaway while porting tests manually. I could even handle it myself if I had permission to configure the system. Let me know your thoughts, Giacomo From: gem5-dev mailto:gem5-dev-boun...@gem5.org>> on behalf of Jason Lowe-Power mailto:ja...@lowepower.com>> Sent: 16 April 2019 16:30 To: gem5 Developer List; Rahul Thakur Subject: [gem5-dev] Continuous integration is live! Hi all, We now have initial support for continuous integration testing! We should all thank Google for donating the CPU time and infrastructure to run these tests. Specifically, Rahul Thakur has been incredibly helpful for the past two years in getting this off the ground. Thanks, Rahul and the rest of the team at Google who has been helping us set this up! Now, if you submit a patch to gerrit and receive a maintainer +1, "kokoro" will kick off a build / test of gem5. Once that is complete, you will receive a verified +1. If it fails, you will receive a verified -1. The logs can be viewed by anyone once the job is completed by following the link posted by kokoro (the https://source.cloud.google.com, not the sponge link). You can see an example on a patch I recently submitted here: https://gem5-review.googlesource.com/c/public/gem5/+/18068. Note that the tests take a couple of hours to run. However, I believe there is no limit to the number of different changes that can be tested at the same time. Soon, we are going to enable commit gating with the verified +1 tag. I.e., you will have to pass the continuous integration tests before you can commit your code. Note that this is using the "new" testing infrastructure. You can run this locally by running "./main.py" in the tests directory. More information about how to run tests and add tests can be found in the TESTING.md file. If there are any questions/issues do not hesitate to contact me or the list. The documentation for the new infrastructure can still be improved. Right now, we're running about 30 tests. You can find the tests that we are running in the tests/gem5 directory. We are looking for volunteers to help us port more of the old tests to the new infrastructure and to expand the coverage of our tests. I'm happy to help anyone get started on this and point out which tests still need to be migrated, where our biggest coverage holes are, etc. Cheers, Jason ___ gem5-dev mailing list gem5-dev@gem5.org<mailto:gem5-dev@gem5.org> http://m5sim.org/mailman/listinfo/gem5-dev IMPORTANT NOTICE: The contents of this email and any attachments are conf
Re: [gem5-dev] Continuous integration is live!
Hi Giacomo, You *do* have permission :). Anyone can modify tests/jenkins/presubmit.cfg and presubmit.sh. In fact, if you look at the history of the presubmit.sh, it *was* running the old tests. See https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, for instance. The problem is that we can't distribute most of the binaries (e.g., SPEC binaries). We could probably upload them to a private location on the Google Cloud and have jenkins consume them that way, but I believe that will be more work than it's worth. I personally believe that putting effort into porting tests is more worth everyone's time than trying to get the old tests to run, but that's just my opinion. I'm happy to merge changes to run the old tests. I personally believe we should only merge tests into the verification tester which everyone can run locally, but I'm open to proprietary tests, especially in the short term if we have a plan to make them not proprietary. Cheers, Jason On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini < giacomo.travagl...@arm.com> wrote: > Hi Jason, > > It's really amazing that we have a testing framework in place, thanks for > your effort! > At the moment as far as I can tell we are only running tests registered > within the new > testing library. > > I was wondering if we could temporarily enable the system to run legacy > quick regressions as well, > while waiting for porting those to the new library. I guess it is > something that shouldn't require a lot of work > > (just calling .util/regress I guess) > > I am saying this since a patch recently merged broke some syscall > emulation tests and I think it would > be beneficial for us to run the entire test suite straightaway while > porting tests manually. > I could even handle it myself if I had permission to configure the system. > > Let me know your thoughts, > > Giacomo > > -- > *From:* gem5-dev on behalf of Jason > Lowe-Power > *Sent:* 16 April 2019 16:30 > *To:* gem5 Developer List; Rahul Thakur > *Subject:* [gem5-dev] Continuous integration is live! > > Hi all, > > We now have initial support for continuous integration testing! We should > all thank Google for donating the CPU time and infrastructure to run these > tests. Specifically, Rahul Thakur has been incredibly helpful for the past > two years in getting this off the ground. Thanks, Rahul and the rest of the > team at Google who has been helping us set this up! > > Now, if you submit a patch to gerrit and receive a maintainer +1, "kokoro" > will kick off a build / test of gem5. Once that is complete, you will > receive a verified +1. If it fails, you will receive a verified -1. The > logs can be viewed by anyone once the job is completed by following the > link posted by kokoro (the https://source.cloud.google.com, not the sponge > link). You can see an example on a patch I recently submitted here: > https://gem5-review.googlesource.com/c/public/gem5/+/18068. Note that the > tests take a couple of hours to run. However, I believe there is no limit > to the number of different changes that can be tested at the same time. > > Soon, we are going to enable commit gating with the verified +1 tag. I.e., > you will have to pass the continuous integration tests before you can > commit your code. > > Note that this is using the "new" testing infrastructure. You can run this > locally by running "./main.py" in the tests directory. More information > about how to run tests and add tests can be found in the TESTING.md file. > If there are any questions/issues do not hesitate to contact me or the > list. The documentation for the new infrastructure can still be improved. > Right now, we're running about 30 tests. You can find the tests that we are > running in the tests/gem5 directory. > > We are looking for volunteers to help us port more of the old tests to the > new infrastructure and to expand the coverage of our tests. I'm happy to > help anyone get started on this and point out which tests still need to be > migrated, where our biggest coverage holes are, etc. > > Cheers, > Jason > ___ > gem5-dev mailing list > gem5-dev@gem5.org > http://m5sim.org/mailman/listinfo/gem5-dev > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] Continuous integration is live!
Hi Jason, It's really amazing that we have a testing framework in place, thanks for your effort! At the moment as far as I can tell we are only running tests registered within the new testing library. I was wondering if we could temporarily enable the system to run legacy quick regressions as well, while waiting for porting those to the new library. I guess it is something that shouldn't require a lot of work (just calling .util/regress I guess) I am saying this since a patch recently merged broke some syscall emulation tests and I think it would be beneficial for us to run the entire test suite straightaway while porting tests manually. I could even handle it myself if I had permission to configure the system. Let me know your thoughts, Giacomo From: gem5-dev on behalf of Jason Lowe-Power Sent: 16 April 2019 16:30 To: gem5 Developer List; Rahul Thakur Subject: [gem5-dev] Continuous integration is live! Hi all, We now have initial support for continuous integration testing! We should all thank Google for donating the CPU time and infrastructure to run these tests. Specifically, Rahul Thakur has been incredibly helpful for the past two years in getting this off the ground. Thanks, Rahul and the rest of the team at Google who has been helping us set this up! Now, if you submit a patch to gerrit and receive a maintainer +1, "kokoro" will kick off a build / test of gem5. Once that is complete, you will receive a verified +1. If it fails, you will receive a verified -1. The logs can be viewed by anyone once the job is completed by following the link posted by kokoro (the https://source.cloud.google.com, not the sponge link). You can see an example on a patch I recently submitted here: https://gem5-review.googlesource.com/c/public/gem5/+/18068. Note that the tests take a couple of hours to run. However, I believe there is no limit to the number of different changes that can be tested at the same time. Soon, we are going to enable commit gating with the verified +1 tag. I.e., you will have to pass the continuous integration tests before you can commit your code. Note that this is using the "new" testing infrastructure. You can run this locally by running "./main.py" in the tests directory. More information about how to run tests and add tests can be found in the TESTING.md file. If there are any questions/issues do not hesitate to contact me or the list. The documentation for the new infrastructure can still be improved. Right now, we're running about 30 tests. You can find the tests that we are running in the tests/gem5 directory. We are looking for volunteers to help us port more of the old tests to the new infrastructure and to expand the coverage of our tests. I'm happy to help anyone get started on this and point out which tests still need to be migrated, where our biggest coverage holes are, etc. Cheers, Jason ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
