Re: RFR: 8323515: Create test alias "all" for all test roots
On Mon, 15 Jan 2024 11:05:09 GMT, Aleksey Shipilev wrote:
> Since recent work to improve `tier4` performance, we actually test
> `tier{1,2,3,4}` often, which includes all the tests in current tree. It would
> be more convenient to just have the `all` test group/alias, so that we can do
> `make test TEST=all`. This also gives a parallelism / run time benefit, as we
> do not wait for tests in each tier to complete before moving to next tier.
>
> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
> environments one also needs to supply a few keywords like `!printer` to skip
> tests that cannot complete without failure due to misconfiguration. I left
> the keywords as is to show how would a failing run look. There is also an
> existing shortcut in build system that allows to run this with `make
> test-all`.
>
>
> % make test TEST=all
>
> Test selection 'all', will run:
> * jtreg:test/hotspot/jtreg:all
> * jtreg:test/jdk:all
> * jtreg:test/langtools:all
> * jtreg:test/jaxp:all
> * jtreg:test/lib-test:all
>
> (...about 6 hours later...)
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>>> jtreg:test/hotspot/jtreg:all 6731 670229 0 <<
>>> jtreg:test/jdk:all 9962 995111 0 <<
>jtreg:test/langtools:all 4469 4469 0 0
>
>jtreg:test/jaxp:all 513 513 0 0
>
>jtreg:test/lib-test:all 3232 0 0
>
> ==
> TEST FAILURE
Okay - change is harmless with no ongoing maintenance cost.
test/jdk/TEST.groups line 28:
> 26: #
> 27:
> 28: all = \
Why no `jdk_all` definition in this case?
-
Marked as reviewed by dholmes (Reviewer).
PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1822313872
PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1452781088
Re: RFR: 8323515: Create test alias "all" for all test roots [v2]
> Since recent work to improve `tier4` performance, we actually test
> `tier{1,2,3,4}` often, which includes all the tests in current tree. It would
> be more convenient to just have the `all` test group/alias, so that we can do
> `make test TEST=all`. This also gives a parallelism / run time benefit, as we
> do not wait for tests in each tier to complete before moving to next tier.
>
> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
> environments one also needs to supply a few keywords like `!printer` to skip
> tests that cannot complete without failure due to misconfiguration. I left
> the keywords as is to show how would a failing run look. There is also an
> existing shortcut in build system that allows to run this with `make
> test-all`.
>
>
> % make test TEST=all
>
> Test selection 'all', will run:
> * jtreg:test/hotspot/jtreg:all
> * jtreg:test/jdk:all
> * jtreg:test/langtools:all
> * jtreg:test/jaxp:all
> * jtreg:test/lib-test:all
>
> (...about 6 hours later...)
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>>> jtreg:test/hotspot/jtreg:all 6731 670229 0 <<
>>> jtreg:test/jdk:all 9962 995111 0 <<
>jtreg:test/langtools:all 4469 4469 0 0
>
>jtreg:test/jaxp:all 513 513 0 0
>
>jtreg:test/lib-test:all 3232 0 0
>
> ==
> TEST FAILURE
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
jdk_all and lib_test_all groups
-
Changes:
- all: https://git.openjdk.org/jdk/pull/17422/files
- new: https://git.openjdk.org/jdk/pull/17422/files/7f6797b6..78f5f9bd
Webrevs:
- full: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=01
- incr: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=00-01
Stats: 8 lines in 2 files changed: 8 ins; 0 del; 0 mod
Patch: https://git.openjdk.org/jdk/pull/17422.diff
Fetch: git fetch https://git.openjdk.org/jdk.git pull/17422/head:pull/17422
PR: https://git.openjdk.org/jdk/pull/17422
Re: RFR: 8323515: Create test alias "all" for all test roots [v2]
On Mon, 15 Jan 2024 22:37:36 GMT, David Holmes wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> jdk_all and lib_test_all groups > > test/jdk/TEST.groups line 28: > >> 26: # >> 27: >> 28: all = \ > > Why no `jdk_all` definition in this case? Tried not to introduce new `*_all` groups here. `jdk_all` would be the same as `jdk:all`, TBH. But we still can do it for symmetry reasons, see new commit. - PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1453098855
Re: RFR: 8323515: Create test alias "all" for all test roots [v2]
On Tue, 16 Jan 2024 08:47:38 GMT, Aleksey Shipilev wrote: >> test/jdk/TEST.groups line 28: >> >>> 26: # >>> 27: >>> 28: all = \ >> >> Why no `jdk_all` definition in this case? > > Tried not to introduce new `*_all` groups here. `jdk_all` would be the same > as `jdk:all`, TBH. But we still can do it for symmetry reasons, see new > commit. "all" looks okay but the comment "Catch-all" suggests something else, shouldn't be "All tests"? - PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1453103766
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
> Since recent work to improve `tier4` performance, we actually test
> `tier{1,2,3,4}` often, which includes all the tests in current tree. It would
> be more convenient to just have the `all` test group/alias, so that we can do
> `make test TEST=all`. This also gives a parallelism / run time benefit, as we
> do not wait for tests in each tier to complete before moving to next tier.
>
> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
> environments one also needs to supply a few keywords like `!printer` to skip
> tests that cannot complete without failure due to misconfiguration. I left
> the keywords as is to show how would a failing run look. There is also an
> existing shortcut in build system that allows to run this with `make
> test-all`.
>
>
> % make test TEST=all
>
> Test selection 'all', will run:
> * jtreg:test/hotspot/jtreg:all
> * jtreg:test/jdk:all
> * jtreg:test/langtools:all
> * jtreg:test/jaxp:all
> * jtreg:test/lib-test:all
>
> (...about 6 hours later...)
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>>> jtreg:test/hotspot/jtreg:all 6731 670229 0 <<
>>> jtreg:test/jdk:all 9962 995111 0 <<
>jtreg:test/langtools:all 4469 4469 0 0
>
>jtreg:test/jaxp:all 513 513 0 0
>
>jtreg:test/lib-test:all 3232 0 0
>
> ==
> TEST FAILURE
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
Catch-all -> All tests
-
Changes:
- all: https://git.openjdk.org/jdk/pull/17422/files
- new: https://git.openjdk.org/jdk/pull/17422/files/78f5f9bd..def2f39b
Webrevs:
- full: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=02
- incr: https://webrevs.openjdk.org/?repo=jdk&pr=17422&range=01-02
Stats: 5 lines in 5 files changed: 0 ins; 0 del; 5 mod
Patch: https://git.openjdk.org/jdk/pull/17422.diff
Fetch: git fetch https://git.openjdk.org/jdk.git pull/17422/head:pull/17422
PR: https://git.openjdk.org/jdk/pull/17422
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 08:52:03 GMT, Alan Bateman wrote: >> Tried not to introduce new `*_all` groups here. `jdk_all` would be the same >> as `jdk:all`, TBH. But we still can do it for symmetry reasons, see new >> commit. > > "all" looks okay but the comment "Catch-all" suggests something else, > shouldn't be "All tests"? Yeah, we can do "All tests" instead. See new commit. - PR Review Comment: https://git.openjdk.org/jdk/pull/17422#discussion_r1453113607
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
Marked as reviewed by alanb (Reviewer).
-
PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1822902993
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
Any other reviews needed for this? Nominally, this changes the test groups in
langtools, so maybe @lahodaj or @biboudis want to take a look. For jaxp,
@JoeWang-Java, maybe?
-
PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1895680732
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
Thanks for the reminder. The new alias is nice, easier to run all tier tests. I
often run xml-only as well that includes jaxp_all plus a small set of jaxp
tests in jdk_all (test/jdk/javax/xml/jaxp). But that's just me, jdk_all already
covers those tests.
-
Marked as reviewed by joehw (Reviewer).
PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1827838226
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
I have nothing against it, but I don't think I know enough about details of
jtreg configuration to provide approval. OTOH, I personally don't see a strong
need for it.
-
PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1905864727
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
All right, thanks! @lmesnik, I realized I forgot to ask if you had objections
to this.
-
PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1906338893
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
Marked as reviewed by lmesnik (Reviewer).
-
PR Review: https://git.openjdk.org/jdk/pull/17422#pullrequestreview-1839361504
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 16 Jan 2024 09:01:35 GMT, Aleksey Shipilev wrote:
>> Since recent work to improve `tier4` performance, we actually test
>> `tier{1,2,3,4}` often, which includes all the tests in current tree. It
>> would be more convenient to just have the `all` test group/alias, so that we
>> can do `make test TEST=all`. This also gives a parallelism / run time
>> benefit, as we do not wait for tests in each tier to complete before moving
>> to next tier.
>>
>> Sample run on out-of-the-box Linux x86_64 fastdebug is below. For some
>> environments one also needs to supply a few keywords like `!printer` to skip
>> tests that cannot complete without failure due to misconfiguration. I left
>> the keywords as is to show how would a failing run look. There is also an
>> existing shortcut in build system that allows to run this with `make
>> test-all`.
>>
>>
>> % make test TEST=all
>>
>> Test selection 'all', will run:
>> * jtreg:test/hotspot/jtreg:all
>> * jtreg:test/jdk:all
>> * jtreg:test/langtools:all
>> * jtreg:test/jaxp:all
>> * jtreg:test/lib-test:all
>>
>> (...about 6 hours later...)
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:all 6731 670229 0
<<
jtreg:test/jdk:all 9962 995111 0
<<
>>jtreg:test/langtools:all 4469 4469 0 0
>>
>>jtreg:test/jaxp:all 513 513 0 0
>>
>>jtreg:test/lib-test:all 3232 0 0
>>
>> ==
>> TEST FAILURE
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Catch-all -> All tests
Thank you all!
-
PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1906520760
Re: RFR: 8323515: Create test alias "all" for all test roots [v3]
On Tue, 23 Jan 2024 17:03:13 GMT, Aleksey Shipilev wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Catch-all -> All tests > > Thank you all! Hi @shipilev , plese see https://github.com/openjdk/jdk/pull/17558 . Thanks. - PR Comment: https://git.openjdk.org/jdk/pull/17422#issuecomment-1908355476
