This is an automated email from the ASF dual-hosted git repository. jungm pushed a commit to branch ee11 in repository https://gitbox.apache.org/repos/asf/tomee-tck.git
commit b8edfbef591e0c35dc01e0e4941142653ba477b0 Author: Markus Jung <[email protected]> AuthorDate: Sun Jul 19 10:17:32 2026 +0200 Run the Faces old-tck-selenium modules in the faces TCK suite Add old-tck-selenium (ajax, commandLink, protectedViews) to the faces invoker run and the expected-module list. Clean each TCK reactor before verify so the aggregating check never counts stale reports left by an earlier run, and count only final surefire/failsafe outcomes so a Chrome test that recovers on rerun no longer reads as an error. With exclusions applied the suite runs 298 tests, 0 failures, 0 errors, 17 skipped; old-tck-selenium needs no exclusions. --- KNOWN_ISSUES.md | 14 +++++--------- runner-standalone/README.md | 2 +- runner-standalone/exclusions/faces.txt | 15 ++++++++------- runner-standalone/faces/pom.xml | 25 +++++++++++++++++-------- runner-standalone/verify-invoker-result.sh | 26 +++++++++++++++++++++----- 5 files changed, 52 insertions(+), 30 deletions(-) diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md index f0e2d51..19d7236 100644 --- a/KNOWN_ISSUES.md +++ b/KNOWN_ISSUES.md @@ -40,7 +40,7 @@ Detail lives next to each runner: | security | 132 tests, 5 F + 2 E; signature test passes | 7 tests | TomEE Jakarta Security | | authentication | 106 tests, 1 F; signature test passes | 1 method (spi `CheckMsgInfoKey`) | TCK challenge #219 (hard-codes a JACC requirement) | | websocket | 737 tests, 3 E | 3 methods | Client container advertises permessage-deflate in the negotiated extension lists | -| faces (modern modules) | 327 tests, 45 F + 6 E (failsafe reruns inflate the counts; 10 distinct failing classes) | 10 classes | 1 TomEE faces-config gap + Mojarra/TomEE CDI-injection, whole-bean/method validation, and one Chrome ajax quirk | +| faces (modern modules + old-tck-selenium) | 298 tests, 0 F + 0 E, 17 skipped (with exclusions applied) | 10 classes | 1 TomEE faces-config gap + Mojarra/TomEE CDI-injection, whole-bean/method validation, and one Chrome ajax quirk | | faces-old (JavaTest) | 5,391 tests, all pass (recorded run: 5 F from a foreign server answering :8080 mid-run; pass on re-run) | — | — (standalone mode, no exclusions) | | faces-signaturetest | passes against Plume's Mojarra (org.glassfish:jakarta.faces 4.1.9) | — | — | @@ -248,14 +248,10 @@ Need triage/fixes in the upstream projects TomEE ships. Not product bugs — gaps in this repository's coverage. - **Modern faces reactor not in CI**: the `faces` runner (modern Arquillian - modules plus the signature test) runs locally via - `run-standalone-suite.sh`; it joins the Jenkins branch list once its - baseline and exclusion wiring have a verified green run. The `faces-old` - JavaTest half runs in CI. -- **Faces `old-tck-selenium` module uncovered**: the extracted TCK reactor - ships an `old-tck-selenium` module (22 tests) that neither the `faces` - runner (which drives faces22/23/40/41 + faces-signaturetest) nor the - `faces-old` JavaTest runner executes; wire it into one of them. + modules, the Chrome/Selenium `old-tck-selenium` modules, and the signature + test) runs locally via `run-standalone-suite.sh`; it joins the Jenkins + branch list once its baseline and exclusion wiring have a verified green + run. The `faces-old` JavaTest half runs in CI. ## What CI runs diff --git a/runner-standalone/README.md b/runner-standalone/README.md index 79f7fd7..13017d0 100644 --- a/runner-standalone/README.md +++ b/runner-standalone/README.md @@ -61,7 +61,7 @@ derived from; with the default exclusions applied these suites run green. | WebSocket 2.2 | EFTL zip (installed as `jakarta.tck:websocket-tck-spec-tests:2.2.0`) | `websocket` | **Runs: 737 tests, 0 failures, 3 errors** (2026-07-18) against Tomcat's WebSocket implementation, driven through the `tomcat-websocket` client container; the signature test passes. The negative-deployment classes run and pass: an invalid server endpoint aborts the whole webapp deployment as the spec requires, and the runner's Arquillian extension (`NegativeDeploymentToleranceObserver`) [...] | Security 4.0 | Source reactor zip 4.0.1 | `security` (Maven module) | **Runs: 26 app modules plus the signature test, 132 tests, 5 failures, 2 errors at baseline; green with the reviewed exclusions (all 27 invoker projects pass, verified end-to-end 2026-07-18)**. The runner downloads and patches the reactor, injects a tomee-remote profile (including the `trustStore.path`/`trustStore.password` properties the OpenID modules' keytool steps read), and drives every module through the Maven [...] | Authentication 3.1 | Source reactor zip 3.1.2 | `authentication` (Maven module) | **Runs: 12 Web Profile modules plus the signature test; green with the reviewed exclusions (all 13 invoker projects pass, 0 failures, verified end-to-end 2026-07-18)**. The spi module registers the TCK's test `AuthConfigProvider` under Tomcat's JASPIC app-context naming (`Catalina/localhost /spitests_servlet_web`), so `ServletProfileSPITest` runs against Tomcat's `AuthConfigFactory` and passes 56 of its 5 [...] -| Faces 4.1 | Source reactor zip 4.1.2 | `faces` (Maven module) | **Runs: modern Arquillian modules (faces22/23/40/41) — 327 tests, 45 failures + 6 errors** (2026-07-18) on Plume's Mojarra; the failsafe rerun mechanism inflates those totals, covering 10 distinct failing classes, all in `exclusions/faces.txt`. One is a TomEE product gap (faces-config.xml unmarshaller rejecting the nonstandard *https* `xsi:schemaLocation`, `Spec1760IT`); the rest are Mojarra/TomEE integration behaviors — F [...] +| Faces 4.1 | Source reactor zip 4.1.2 | `faces` (Maven module) | **Runs: modern Arquillian modules (faces22/23/40/41) plus the Chrome/Selenium `old-tck-selenium` modules (ajax, commandLink, protectedViews) — 298 tests, 0 failures, 0 errors, 17 skipped** on Plume's Mojarra, with `exclusions/faces.txt` applied. The 10 excluded classes are one TomEE product gap (faces-config.xml unmarshaller rejecting the nonstandard *https* `xsi:schemaLocation`, `Spec1760IT`) and Mojarra/TomEE integration [...] | Faces 4.1 (legacy old-tck) | Source reactor zip 4.1.2 (old-tck built from bundled sources) | `faces-old` (Maven module) | **Passes 5,391/5,391 (standalone mode, no exclusions)** (2026-07-18). The recorded full run shows 5 failures in `htmloutcometargetbutton`, all from a foreign server answering port 8080 mid-run; they pass on re-run against TomEE. The JavaTest half deploys through the TCK's own `tomcat` handler (WAR copy into `webapps/`) against a TomEE instance the runner provisions [...] | JSON Processing 2.1 | `jakarta.json:jakarta.json-tck-*:2.1.1` (Maven Central; byte-identical to the EFTL zip pinned in `environment/versions.env`) | `jsonp` | **Passes 197/197** (2026-07-18) against Apache Johnzon 2.1.0, the JSON-P provider bundled in the TomEE Plume snapshot: 179 functional + signature tests and 18 pluggability tests. The signature test checks the `jakarta.json` packages of the distribution's `jakartaee-api` jar | | JSON Binding 3.0 | `jakarta.json.bind:jakarta.json.bind-tck:3.0.0` (Maven Central; byte-identical to the EFTL zip pinned in `environment/versions.env`) | `jsonb` | **Runs: 295 tests, 1 failure, 1 error, 5 skipped** against Apache Johnzon 2.1.0 with OpenWebBeans as the CDI SE container (the runner boots it because the TCK's own private `@BeforeAll` bootstrap is ignored by JUnit). The runner sets `johnzon.use-bigdecimal-stringadapter=false`/`johnzon.use-biginteger-stringadapter=false` (J [...] diff --git a/runner-standalone/exclusions/faces.txt b/runner-standalone/exclusions/faces.txt index 54efc8d..11f5944 100644 --- a/runner-standalone/exclusions/faces.txt +++ b/runner-standalone/exclusions/faces.txt @@ -1,10 +1,10 @@ # Jakarta Faces 4.1 TCK source reactor (4.1.2), modern Arquillian modules -# (faces22/23/40/41 + faces-signaturetest) on TomEE Plume's Mojarra, Java 21, -# 2026-07-18. Full baseline without this file: 327 tests, 45 failures + 6 -# errors, 5 skipped (the failsafe rerun mechanism counts each retried method -# up to five times, so these totals cover 10 distinct failing test classes). -# The excluded work splits into one TomEE product gap and four Mojarra/TomEE -# integration behaviors: +# (faces22/23/40/41), the Chrome/Selenium old-tck-selenium modules (ajax, +# commandLink, protectedViews), and faces-signaturetest on TomEE Plume's +# Mojarra, Java 21. With this file applied the suite runs 298 tests, 0 +# failures, 0 errors, 17 skipped. The 10 classes excluded below split into +# one TomEE product gap and four Mojarra/TomEE integration behaviors; +# old-tck-selenium runs unmodified and needs no exclusions: # - faces-config xsi:schemaLocation on the nonstandard https XSI namespace: # 1 class (Spec1760IT) whose faces-config carries child elements. # - Faces CDI implicit-object injection (@Inject @RequestCookieMap / @@ -15,7 +15,8 @@ # - Mojarra ajax re-init under Chrome (flagged in the TCK source itself): 1 # class. # The signature test and the rest of the reactor pass. The legacy JavaTest -# old-tck half lives in runner-standalone-faces-old and is not covered here. +# old-tck half (a separate JavaTest harness) lives in +# runner-standalone-faces-old and is not covered here. # # Format: maven-surefire/failsafe excludesFile patterns; the runner passes # this file to every TCK module through the invoker as diff --git a/runner-standalone/faces/pom.xml b/runner-standalone/faces/pom.xml index a2daf43..47be7dd 100644 --- a/runner-standalone/faces/pom.xml +++ b/runner-standalone/faces/pom.xml @@ -18,15 +18,17 @@ <artifactId>runner-standalone-faces</artifactId> <packaging>pom</packaging> - <name>Jakarta Faces 4.1 TCK runner (modern modules + signature test)</name> + <name>Jakarta Faces 4.1 TCK runner (modern modules + old-tck-selenium + signature test)</name> <description> Runs the modern Arquillian modules of the Jakarta Faces 4.1 TCK - (faces22/23/40/41 and util, roughly 319 tests) plus the - faces-signaturetest module against TomEE Plume, which ships Mojarra. - The signature test checks the Mojarra artifact TomEE bundles - (faces.impl.version below must track the jakarta.faces jar in the - distribution's lib directory). The legacy JavaTest old-tck half lives in - runner-standalone-faces-old. Run with: run-standalone-suite.sh faces + (faces22/23/40/41 and util) plus the Chrome/Selenium old-tck-selenium + modules (ajax, commandLink, protectedViews) and the faces-signaturetest + module against TomEE Plume, which ships Mojarra. With exclusions applied + the suite runs 298 tests. The signature test checks the Mojarra artifact + TomEE bundles (faces.impl.version below must track the jakarta.faces jar + in the distribution's lib directory). The legacy JavaTest old-tck half + lives in runner-standalone-faces-old. Run with: + run-standalone-suite.sh faces </description> <properties> @@ -147,6 +149,7 @@ <argument>faces23</argument> <argument>faces40</argument> <argument>faces41</argument> + <argument>old-tck-selenium</argument> <argument>faces-signaturetest</argument> </arguments> </configuration> @@ -173,7 +176,12 @@ <phase>integration-test</phase> <goals><goal>run</goal></goals> <configuration> - <goals><goal>verify</goal></goals> + <!-- clean before verify so the aggregating verify step never + counts stale surefire/failsafe reports left in a module's + target/ by an earlier run (e.g. a full-baseline pass whose + now-excluded classes no longer re-run and overwrite their + reports). --> + <goals><goal>clean</goal><goal>verify</goal></goals> <profiles><profile>tomee-remote</profile></profiles> <!-- Each include is a multi-module reactor; collect the whole baseline instead of stopping at the first failing module. @@ -189,6 +197,7 @@ <pomInclude>faces23/pom.xml</pomInclude> <pomInclude>faces40/pom.xml</pomInclude> <pomInclude>faces41/pom.xml</pomInclude> + <pomInclude>old-tck-selenium/pom.xml</pomInclude> <pomInclude>faces-signaturetest/pom.xml</pomInclude> </pomIncludes> </configuration> diff --git a/runner-standalone/verify-invoker-result.sh b/runner-standalone/verify-invoker-result.sh index eb7169e..c816023 100755 --- a/runner-standalone/verify-invoker-result.sh +++ b/runner-standalone/verify-invoker-result.sh @@ -43,18 +43,34 @@ fail() { } # --- Test failures and errors across every raw report under the reactor. --- -# TEST-*.xml carries failures="" errors="" on the <testsuite> element; -# failsafe-summary.xml carries <failures>/<errors> (and records deployment -# errors the build log does not surface as a "Tests run" line). +# Count only *final* outcomes. A <testcase> that failed but passed on a +# failsafe/surefire rerun records its failed attempt as a <flakyError> or +# <flakyFailure> (and the run reports <flakes> in failsafe-summary.xml); the +# <testsuite> failures="" errors="" attributes still tally those recovered +# attempts, so summing the attributes turns a green flaky test red. Count the +# real <failure>/<error> elements that hang directly off a <testcase> instead, +# which excludes the recovered flaky/rerun attempts. report_failures=0 report_errors=0 for report in $(find "$reactor_root" -name 'TEST-*.xml' -path '*-reports/*' 2>/dev/null); do - f=$(sed -n 's/.*<testsuite[^>]* failures="\([0-9]*\)".*/\1/p' "$report" | head -1) - e=$(sed -n 's/.*<testsuite[^>]* errors="\([0-9]*\)".*/\1/p' "$report" | head -1) + # grep -c prints the count and exits non-zero on no match; strip any newline + # and default to 0 so the arithmetic stays single-line. <error and <failure + # do not match the recovered <flakyError>/<rerunError>/<flakyFailure>/ + # <rerunFailure> attempts (their tag name is not preceded by '<error'/ + # '<failure'). + f=$(grep -c '<failure' "$report" 2>/dev/null | head -1) + e=$(grep -c '<error' "$report" 2>/dev/null | head -1) report_failures=$((report_failures + ${f:-0})) report_errors=$((report_errors + ${e:-0})) done +# failsafe-summary.xml records a deployment error the build log never surfaces +# as a "Tests run" line and that writes no TEST-*.xml; count those summaries +# only, so the per-testcase tally above is not double-counted. for summary in $(find "$reactor_root" -name 'failsafe-summary.xml' 2>/dev/null); do + reports_dir=$(dirname "$summary") + if ls "$reports_dir"/TEST-*.xml >/dev/null 2>&1; then + continue + fi f=$(sed -n 's/.*<failures>\([0-9]*\)<.*/\1/p' "$summary" | head -1) e=$(sed -n 's/.*<errors>\([0-9]*\)<.*/\1/p' "$summary" | head -1) report_failures=$((report_failures + ${f:-0}))
