Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Mon, 15 Jan 2024 10:48:23 GMT, Aleksey Shipilev wrote: > Some jtreg tests require resolvable external dependencies. This resolution is > delegated to JIB, which is not used in vanilla OpenJDK testing. It would be > convenient to add a keyword that marks tests that require these external > dependencies, so that we could exclude those tests from runs. This would > allow us to: a) run all tests in hotspot:tier4, which now excludes > `applications/` specifically; b) make all tests runs (#17422) cleaner on many > environments. > > I provisionally call this flag `external-dep`, but I am open for other > suggestions. > > Note that some tests that pull `@Artifact`-s provide special paths that do > limited testing anyway. However, there are tests which cannot run without > external dependencies at all. These include at least `applications/jcstress` > and `applications/scimark` tests. > > Ironically, I cannot run the jcstress test generator because the dependencies > are lacking here. I regenerated those test using a self-built jcstress 0.16 > bundle. > > Additional testing: > - [x] `make test TEST=applications/` fails > - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, > skipping most of the tests Awesome, thanks. - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1910723514
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Wed, 24 Jan 2024 21:28:29 GMT, Leonid Mesnik wrote: >> Some jtreg tests require resolvable external dependencies. This resolution >> is delegated to JIB, which is not used in vanilla OpenJDK testing. It would >> be convenient to add a keyword that marks tests that require these external >> dependencies, so that we could exclude those tests from runs. This would >> allow us to: a) run all tests in hotspot:tier4, which now excludes >> `applications/` specifically; b) make all tests runs (#17422) cleaner on >> many environments. >> >> I provisionally call this flag `external-dep`, but I am open for other >> suggestions. >> >> Note that some tests that pull `@Artifact`-s provide special paths that do >> limited testing anyway. However, there are tests which cannot run without >> external dependencies at all. These include at least `applications/jcstress` >> and `applications/scimark` tests. >> >> Ironically, I cannot run the jcstress test generator because the >> dependencies are lacking here. I regenerated those test using a self-built >> jcstress 0.16 bundle. >> >> Additional testing: >> - [x] `make test TEST=applications/` fails >> - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, >> skipping most of the tests > > Marked as reviewed by lmesnik (Reviewer). > @lmesnik, you good with the keyword name? Yes, I'm fine. - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1910613276
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Wed, 24 Jan 2024 21:28:29 GMT, Leonid Mesnik wrote: >> Some jtreg tests require resolvable external dependencies. This resolution >> is delegated to JIB, which is not used in vanilla OpenJDK testing. It would >> be convenient to add a keyword that marks tests that require these external >> dependencies, so that we could exclude those tests from runs. This would >> allow us to: a) run all tests in hotspot:tier4, which now excludes >> `applications/` specifically; b) make all tests runs (#17422) cleaner on >> many environments. >> >> I provisionally call this flag `external-dep`, but I am open for other >> suggestions. >> >> Note that some tests that pull `@Artifact`-s provide special paths that do >> limited testing anyway. However, there are tests which cannot run without >> external dependencies at all. These include at least `applications/jcstress` >> and `applications/scimark` tests. >> >> Ironically, I cannot run the jcstress test generator because the >> dependencies are lacking here. I regenerated those test using a self-built >> jcstress 0.16 bundle. >> >> Additional testing: >> - [x] `make test TEST=applications/` fails >> - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, >> skipping most of the tests > > Marked as reviewed by lmesnik (Reviewer). @lmesnik, you good with the keyword name? - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1909740587
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Mon, 15 Jan 2024 10:48:23 GMT, Aleksey Shipilev wrote: > Some jtreg tests require resolvable external dependencies. This resolution is > delegated to JIB, which is not used in vanilla OpenJDK testing. It would be > convenient to add a keyword that marks tests that require these external > dependencies, so that we could exclude those tests from runs. This would > allow us to: a) run all tests in hotspot:tier4, which now excludes > `applications/` specifically; b) make all tests runs (#17422) cleaner on many > environments. > > I provisionally call this flag `external-dep`, but I am open for other > suggestions. > > Note that some tests that pull `@Artifact`-s provide special paths that do > limited testing anyway. However, there are tests which cannot run without > external dependencies at all. These include at least `applications/jcstress` > and `applications/scimark` tests. > > Ironically, I cannot run the jcstress test generator because the dependencies > are lacking here. I regenerated those test using a self-built jcstress 0.16 > bundle. > > Additional testing: > - [x] `make test TEST=applications/` fails > - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, > skipping most of the tests Marked as reviewed by lmesnik (Reviewer). - PR Review: https://git.openjdk.org/jdk/pull/17421#pullrequestreview-1842419609
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Tue, 23 Jan 2024 14:42:20 GMT, Roger Riggs wrote: > Is there any place to document the new keyword or its usage; it does not seem > very discoverable just existing in the TEST.ROOT and some tests. I don't think there is a place to describe keywords, except in the relevant `TEST.ROOT`-s. - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1906324934
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Mon, 15 Jan 2024 10:48:23 GMT, Aleksey Shipilev wrote: > Some jtreg tests require resolvable external dependencies. This resolution is > delegated to JIB, which is not used in vanilla OpenJDK testing. It would be > convenient to add a keyword that marks tests that require these external > dependencies, so that we could exclude those tests from runs. This would > allow us to: a) run all tests in hotspot:tier4, which now excludes > `applications/` specifically; b) make all tests runs (#17422) cleaner on many > environments. > > I provisionally call this flag `external-dep`, but I am open for other > suggestions. > > Note that some tests that pull `@Artifact`-s provide special paths that do > limited testing anyway. However, there are tests which cannot run without > external dependencies at all. These include at least `applications/jcstress` > and `applications/scimark` tests. > > Ironically, I cannot run the jcstress test generator because the dependencies > are lacking here. I regenerated those test using a self-built jcstress 0.16 > bundle. > > Additional testing: > - [x] `make test TEST=applications/` fails > - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, > skipping most of the tests Is there any place to document the new keyword or its usage; it does not seem very discoverable just existing in the TEST.ROOT and some tests. - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1906198126
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Tue, 23 Jan 2024 01:32:45 GMT, David Holmes wrote: > Seems quite reasonable. Thanks! I shall wait for more reviewers, in case someone has an issue with `external-dep` as the flag name. - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1905719123
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Mon, 15 Jan 2024 10:48:23 GMT, Aleksey Shipilev wrote: > Some jtreg tests require resolvable external dependencies. This resolution is > delegated to JIB, which is not used in vanilla OpenJDK testing. It would be > convenient to add a keyword that marks tests that require these external > dependencies, so that we could exclude those tests from runs. This would > allow us to: a) run all tests in hotspot:tier4, which now excludes > `applications/` specifically; b) make all tests runs (#17422) cleaner on many > environments. > > I provisionally call this flag `external-dep`, but I am open for other > suggestions. > > Note that some tests that pull `@Artifact`-s provide special paths that do > limited testing anyway. However, there are tests which cannot run without > external dependencies at all. These include at least `applications/jcstress` > and `applications/scimark` tests. > > Ironically, I cannot run the jcstress test generator because the dependencies > are lacking here. I regenerated those test using a self-built jcstress 0.16 > bundle. > > Additional testing: > - [x] `make test TEST=applications/` fails > - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, > skipping most of the tests Seems quite reasonable. - Marked as reviewed by dholmes (Reviewer). PR Review: https://git.openjdk.org/jdk/pull/17421#pullrequestreview-1837713758
Re: RFR: 8323717: Introduce test keyword for tests that need external dependencies
On Mon, 15 Jan 2024 10:48:23 GMT, Aleksey Shipilev wrote: > Some jtreg tests require resolvable external dependencies. This resolution is > delegated to JIB, which is not used in vanilla OpenJDK testing. It would be > convenient to add a keyword that marks tests that require these external > dependencies, so that we could exclude those tests from runs. This would > allow us to: a) run all tests in hotspot:tier4, which now excludes > `applications/` specifically; b) make all tests runs (#17422) cleaner on many > environments. > > I provisionally call this flag `external-dep`, but I am open for other > suggestions. > > Note that some tests that pull `@Artifact`-s provide special paths that do > limited testing anyway. However, there are tests which cannot run without > external dependencies at all. These include at least `applications/jcstress` > and `applications/scimark` tests. > > Ironically, I cannot run the jcstress test generator because the dependencies > are lacking here. I regenerated those test using a self-built jcstress 0.16 > bundle. > > Additional testing: > - [x] `make test TEST=applications/` fails > - [x] `JTREG_KEYWORDS=!external-dep make test TEST=applications/` passes, > skipping most of the tests Any takers? Maybe the audience should include core-libs too. - PR Comment: https://git.openjdk.org/jdk/pull/17421#issuecomment-1903486053
