Re: RFR: 8323515: Create test alias "all" for all test roots
On Mon, 15 Jan 2024 11:05:09 GMT, Aleksey Shipilev wrote: > Since recent work to improve `tier4` performance, we actually test > `tier{1,2,3,4}` often, which includes all the tests in current tree. It would > be more convenient to just have the `all` test group/alias, so that we can do > `make test TEST=all`. This also gives a parallelism / run time benefit, as we > do not wait for tests in each tier to complete before moving to next tier. > > Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some > environments one also needs to supply a few keywords like `!printer` to skip > tests that cannot complete without failure due to misconfiguration. I left > the keywords as is to show how would a failing run look. There is also an > existing shortcut in build system that allows to run this with `make > test-all`. > > > % make test TEST=all > > Test selection 'all', will run: > * jtreg:test/hotspot/jtreg:all > * jtreg:test/jdk:all > * jtreg:test/langtools:all > * jtreg:test/jaxp:all > * jtreg:test/lib-test:all > > (...about 6 hours later...) > > == > Test summary > == >TEST TOTAL PASS FAIL ERROR > >>> jtreg:test/hotspot/jtreg:all 6731 670229 0 << >>> jtreg:test/jdk:all 9962 995111 0 << >jtreg:test/langtools:all 4469 4469 0 0 > >jtreg:test/jaxp:all 513 513 0 0 > >jtreg:test/lib-test:all 3232 0 0 > > == > TEST FAILURE Okay - change is harmless with no ongoing maintenance cost. test/jdk/TEST.groups line 28: > 26: # > 27: > 28: all = \ Why no `jdk_all` definition in this case? - Marked as reviewed by dholmes (Reviewer). PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1822313872 PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1452781088
Re: RFR: 8323515: Create test alias "all" for all test roots [v2]
> Since recent work to improve `tier4` performance, we actually test > `tier{1,2,3,4}` often, which includes all the tests in current tree. It would > be more convenient to just have the `all` test group/alias, so that we can do > `make test TEST=all`. This also gives a parallelism / run time benefit, as we > do not wait for tests in each tier to complete before moving to next tier. > > Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some > environments one also needs to supply a few keywords like `!printer` to skip > tests that cannot complete without failure due to misconfiguration. I left > the keywords as is to show how would a failing run look. There is also an > existing shortcut in build system that allows to run this with `make > test-all`. > > > % make test TEST=all > > Test selection 'all', will run: > * jtreg:test/hotspot/jtreg:all > * jtreg:test/jdk:all > * jtreg:test/langtools:all > * jtreg:test/jaxp:all > * jtreg:test/lib-test:all > > (...about 6 hours later...) > > == > Test summary > == >TEST TOTAL PASS FAIL ERROR > >>> jtreg:test/hotspot/jtreg:all 6731 670229 0 << >>> jtreg:test/jdk:all 9962 995111 0 << >jtreg:test/langtools:all 4469 4469 0 0 > >jtreg:test/jaxp:all 513 513 0 0 > >jtreg:test/lib-test:all 3232 0 0 > > == > TEST FAILURE Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision: jdk_all and lib_test_all groups - Changes: - all: https://git.openjdk.org/jdk/pull/17422/files - new: https://git.openjdk.org/jdk/pull/17422/files/7f6797b6..78f5f9bd Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=00-01 Stats: 8 lines in 2 files changed: 8 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jdk/pull/17422.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/17422/head:pull/17422 PR: https://git.openjdk.org/jdk/pull/17422
Re: RFR: 8323515: Create test alias "all" for all test roots [v2]
On Mon, 15 Jan 2024 22:37:36 GMT, David Holmes wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> jdk_all and lib_test_all groups > > test/jdk/TEST.groups line 28: > >> 26: # >> 27: >> 28: all = \ > > Why no `jdk_all` definition in this case? Tried not to introduce new `*_all` groups here. `jdk_all` would be the same as `jdk:all`, TBH. But we still can do it for symmetry reasons, see new commit. - PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1453098855
Re: RFR: 8323515: Create test alias "all" for all test roots [v2]
On Tue, 16 Jan 2024 08:47:38 GMT, Aleksey Shipilev wrote: >> test/jdk/TEST.groups line 28: >> >>> 26: # >>> 27: >>> 28: all = \ >> >> Why no `jdk_all` definition in this case? > > Tried not to introduce new `*_all` groups here. `jdk_all` would be the same > as `jdk:all`, TBH. But we still can do it for symmetry reasons, see new > commit. "all" looks okay but the comment "Catch-all" suggests something else, shouldn't be "All tests"? - PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1453103766
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
> Since recent work to improve `tier4` performance, we actually test > `tier{1,2,3,4}` often, which includes all the tests in current tree. It would > be more convenient to just have the `all` test group/alias, so that we can do > `make test TEST=all`. This also gives a parallelism / run time benefit, as we > do not wait for tests in each tier to complete before moving to next tier. > > Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some > environments one also needs to supply a few keywords like `!printer` to skip > tests that cannot complete without failure due to misconfiguration. I left > the keywords as is to show how would a failing run look. There is also an > existing shortcut in build system that allows to run this with `make > test-all`. > > > % make test TEST=all > > Test selection 'all', will run: > * jtreg:test/hotspot/jtreg:all > * jtreg:test/jdk:all > * jtreg:test/langtools:all > * jtreg:test/jaxp:all > * jtreg:test/lib-test:all > > (...about 6 hours later...) > > == > Test summary > == >TEST TOTAL PASS FAIL ERROR > >>> jtreg:test/hotspot/jtreg:all 6731 670229 0 << >>> jtreg:test/jdk:all 9962 995111 0 << >jtreg:test/langtools:all 4469 4469 0 0 > >jtreg:test/jaxp:all 513 513 0 0 > >jtreg:test/lib-test:all 3232 0 0 > > == > TEST FAILURE Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision: Catch-all -> All tests - Changes: - all: https://git.openjdk.org/jdk/pull/17422/files - new: https://git.openjdk.org/jdk/pull/17422/files/78f5f9bd..def2f39b Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=01-02 Stats: 5 lines in 5 files changed: 0 ins; 0 del; 5 mod Patch: https://git.openjdk.org/jdk/pull/17422.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/17422/head:pull/17422 PR: https://git.openjdk.org/jdk/pull/17422
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 08:52:03 GMT, Alan Bateman wrote: >> Tried not to introduce new `*_all` groups here. `jdk_all` would be the same >> as `jdk:all`, TBH. But we still can do it for symmetry reasons, see new >> commit. > > "all" looks okay but the comment "Catch-all" suggests something else, > shouldn't be "All tests"? Yeah, we can do "All tests" instead. See new commit. - PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1453113607
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests Marked as reviewed by alanb (Reviewer). - PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1822902993
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests Any other reviews needed for this? Nominally, this changes the test groups in langtools, so maybe @lahodaj or @biboudis want to take a look. For jaxp, @JoeWang-Java, maybe? - PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1895680732
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests Thanks for the reminder. The new alias is nice, easier to run all tier tests. I often run xml-only as well that includes jaxp_all plus a small set of jaxp tests in jdk_all (test/jdk/javax/xml/jaxp). But that's just me, jdk_all already covers those tests. - Marked as reviewed by joehw (Reviewer). PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1827838226
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests I have nothing against it, but I don't think I know enough about details of jtreg configuration to provide approval. OTOH, I personally don't see a strong need for it. - PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1905864727
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests All right, thanks! @lmesnik, I realized I forgot to ask if you had objections to this. - PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1906338893
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests Marked as reviewed by lmesnik (Reviewer). - PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1839361504
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote: >> Since recent work to improve `tier4` performance, we actually test >> `tier{1,2,3,4}` often, which includes all the tests in current tree. It >> would be more convenient to just have the `all` test group/alias, so that we >> can do `make test TEST=all`. This also gives a parallelism / run time >> benefit, as we do not wait for tests in each tier to complete before moving >> to next tier. >> >> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some >> environments one also needs to supply a few keywords like `!printer` to skip >> tests that cannot complete without failure due to misconfiguration. I left >> the keywords as is to show how would a failing run look. There is also an >> existing shortcut in build system that allows to run this with `make >> test-all`. >> >> >> % make test TEST=all >> >> Test selection 'all', will run: >> * jtreg:test/hotspot/jtreg:all >> * jtreg:test/jdk:all >> * jtreg:test/langtools:all >> * jtreg:test/jaxp:all >> * jtreg:test/lib-test:all >> >> (...about 6 hours later...) >> >> == >> Test summary >> == >>TEST TOTAL PASS FAIL ERROR >> jtreg:test/hotspot/jtreg:all 6731 670229 0 << jtreg:test/jdk:all 9962 995111 0 << >>jtreg:test/langtools:all 4469 4469 0 0 >> >>jtreg:test/jaxp:all 513 513 0 0 >> >>jtreg:test/lib-test:all 3232 0 0 >> >> == >> TEST FAILURE > > Aleksey Shipilev has updated the pull request incrementally with one > additional commit since the last revision: > > Catch-all -> All tests Thank you all! - PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1906520760
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 23 Jan 2024 17:03:13 GMT, Aleksey Shipilev wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Catch-all -> All tests > > Thank you all! Hi @shipilev , plese see https://github.com/openjdk/jdk/pull/17558 . Thanks. - PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1908355476