Hi, I’ve been running individual tests/small groups of tests on my machine, also means I did not run into any of the localization issues Richard mentioned. I think tomee-plume is likely to be more compliant (probably because of eclipselink vs openjpa), AFAIK all certifications beforehand have been done on plume as well.
What I do to run tests is usually: $ mvn clean $ ./runtests --web tomee-plume com/sun/ts/tests/jsonp/pluggability/jsonprovidertests/Client#jsonProviderTest3_from_appclient And then examine logs and debug tomee using -ds, I didn’t need any other special flags yet. I think —ee91 used to activate a profile in the maven build that I removed in 8deda1e1, so we can likely just drop this CLI flag and focus on EE 10. @Paulo, maybe you can share which test have been failing for you (e.g. through a gist)? Also how did you run them? What does your system look like? (Java version, Locale, Linux/OSX/Windows, etc.) Thanks Markus > On 7. Aug 2024, at 20:27, Richard Zowalla <r...@apache.org> wrote: > > Hi, > > First of all: It is really appreciated, that you jumped into exploring the > TCK setup :) > > I think, it might depend on the way you are running those tests. Typically, > it it not sufficient to just run the *.sh file of > https://github.com/apache/tomee-tck/ (at least from my experience). > On my German localized system, I wasn’t able to get some tests working, which > were fine under the https://tck.work/tomee/projects environment. > > A typical thing for some JAX-RS tests with plume looked like: > > rm -rf target/ && ./runtests -Dhttps.protocols=TLSv1.1,TLSv1.2 --ee91 -j > --env -nc -c -U -ol -w tomee-plume > com.sun.ts.tests.jaxrs.api.rs.ext.interceptor.reader.readerinterceptorcontext.JAXRSClient > \ > com.sun.ts.tests.jaxrs.ee.rs.core.responsebuilder.JAXRSClient \ > com.sun.ts.tests.jaxrs.ee.rs.ext.interceptor.containerreader.readerinterceptorcontext.JAXRSClient > \ > com.sun.ts.tests.jaxrs.ee.rs.pathparam.locator.JAXRSLocatorClient \ > com.sun.ts.tests.jaxrs.spec.client.typedentities.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.context.server.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.filter.interceptor.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.filter.lastvalue.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.provider.reader.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.provider.standardnotnull.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.resource.requestmatching.JAXRSClient \ > com.sun.ts.tests.jaxrs.spec.resourceconstructor.JAXRSClient > > > > I guess that we might need to update the „—ee91“ flag to respect EE10, but > didn’t test it myself at the moment. Maybe Markus has some more insights as > (I think) he was able to run some EE10 TCK test lately. > The POM (spec versions) and README might need an update too, so more people > can have the possibility to set things up. > > To try to your questions: > > (1) Share your actual run configuration. How do you start the tests? Which > tests? For a first try, maybe focus on one single test first. > (2) Tomitribe provided tck.work to run the TCK on AWS. I don’t think, that > the current setting is runnable. Maybe JL or Jon have insights here? Are we > able to do a run with a M3-SNAPSHOT? > (3) I would suggest to use a gist to share the logs of a single TCK test. > > From a quick search in the TCK setup, it seems, that openejb.xml uses > javax.jms instead of jakarta.jms (and we are on amq6, so should be jakarta, I > guess - maybe try a search / replace for it?). Might be worth a try, if the > connection factory and related services come up afterwards? > > > > Hope it helps. > Richard > >> Am 05.08.2024 um 14:14 schrieb Paulo Araújo <cris...@vikeo.com.br.INVALID>: >> >> I've began to run the tck-tests on tomee-plume and it seems really bad, so >> I've switched to test tomee-web and still waiting results. It seems there is >> a problem with JMS configuration on tomee test project (all tests fail after >> 10 retries, each one timing-out). >> >> 1) Can someone with some experience guide me how to fix such issues on tests? >> >> 2) Is there any automated environment where these tests take place and are >> reported and, if so, how can I access them? >> >> 3) How can I share the log results (~200k/test group) in the list? >> >> >> >> Thanks in advance, >> >> Paulo >> >> >> >> >> >> >> >> >> >> ---- Em qui, 25 jul 2024 11:12:31 -0300 Richard Zowalla >> <rich...@zowalla.com> escreveu --- >> >> >> >> Standalone is here: https://github.com/apache/tomee/tree/main/tck >> >> As long as it is arquillian/junit/testng based and can be executed via >> Maven. For some stuff, Jiras may exist. >> >> For the tomee-tck itself you don't need to create a Jira imho. We may need >> them for standalone and fixing tests. >> >> Gruß >> Richard >> >> Am 25. Juli 2024 16:05:53 MESZ schrieb "Paulo Araújo" >> <mailto:cris...@vikeo.com.br.INVALID>: >>> Hi, >>> >>> >>> >>> Considering the report of glassfish-7 web profile certification, tomee-10 >>> should pass the following groups of tests: >>> >>> appclient >>> >>> assembly >>> >>> connector >>> >>> ejb >>> >>> ejb30.assembly >>> >>> ejb30.bb >>> >>> ejb30.lite.appexception >>> >>> ejb30.lite.async >>> >>> ejb30.lite.basic >>> >>> ejb30.lite.ejbcontext >>> >>> ejb30.lite.enventry >>> >>> ejb30.lite.interceptor >>> >>> ejb30.lite.lookup >>> >>> ejb30.lite.naming >>> >>> ejb30.lite.nointerface >>> >>> ejb30.lite.packaging >>> >>> ejb30.lite.singleton >>> >>> ejb30.lite.stateful >>> >>> ejb30.lite.tx >>> >>> ejb30.lite.view >>> >>> ejb30.lite.xmloverride >>> >>> ejb30.misc >>> >>> ejb30.sec >>> >>> ejb30.timer >>> >>> ejb30.webservice >>> >>> ejb30.zombie >>> >>> ejb32 >>> >>> el >>> >>> integration >>> >>> jacc >>> >>> javaee >>> >>> javamail >>> >>> jaxrs >>> >>> jaxrs >>> >>> jdbc.ee.batchUpdate >>> >>> jdbc.ee.callStmt >>> >>> jdbc.ee.connection >>> >>> jdbc.ee.dateTime >>> >>> jdbc.ee.dbMeta >>> >>> jdbc.ee.escapeSyntax >>> >>> jdbc.ee.exception >>> >>> jdbc.ee.prepStmt >>> >>> jdbc.ee.resultSet >>> >>> jdbc.ee.rsMeta >>> >>> jdbc.ee.stmt >>> >>> jms >>> >>> jsp >>> >>> jstl >>> >>> jta >>> >>> samples >>> >>> servlet >>> >>> signaturetest >>> >>> webservices12 >>> >>> webservices13 >>> >>> websocket >>> >>> xa >>> >>> >>> >>> I've updated the webprofile.sh of tomee-tck project to reflect these >>> current required tests (pull request of changes will be submitted later, if >>> my assumptions here are confirmed). >>> >>> >>> >>> As I understand from glassfish results, jsf tests were dropped from TCK-10 >>> in favor of individual Faces TCK[1], situation in which are also >>> concurrency [2], json-p [3], json-b [4], jax-rs [5], security [6], >>> authentication [7], CDI [8], validation [9], dependency injection [10] and >>> debugging support for other languages [11]. >>> >>> >>> >>> Previous web profiile certification of tomee [12] had already used this >>> model concerting DI, CDI, validation, XML binding and debugging. >>> >>> The new specs to be covered individually where added by the jakartaee-10 >>> webprofile spec [13]. >>> >>> >>> >>> Does the tomme-tck project is also responsible for running or generating >>> tests results for the individual project? Do I need to open a new Jira >>> Issue to submit the proposed changes to tomee-tck project? >>> >>> >>> >>> Best regards, >>> >>> Paulo >>> >>> >>> >>> >>> >>> [1] >>> https://download.eclipse.org/jakartaee/faces/4.0/jakarta-faces-tck-4.0.1.zip >>> >>> >>> [2] >>> https://download.eclipse.org/jakartaee/concurrency/3.0/concurrency-tck-3.0.2.zip >>> >>> >>> [3] >>> https://download.eclipse.org/jakartaee/jsonp/2.1/jakarta-jsonp-tck-2.1.0.zip >>> >>> >>> [4] >>> https://download.eclipse.org/jakartaee/jsonp/2.1/jakarta-jsonp-tck-2.1.0.zip >>> >>> >>> [5] >>> https://download.eclipse.org/jakartaee/restful-ws/3.1/jakarta-restful-ws-tck-3.1.0.zip >>> >>> >>> [6] >>> https://download.eclipse.org/jakartaee/security/3.0/jakarta-security-tck-3.0.0.zip >>> >>> >>> [7] >>> https://download.eclipse.org/jakartaee/authentication/3.0/jakarta-authentication-tck-3.0.2.zip >>> >>> >>> [8] https://download.eclipse.org/jakartaee/cdi/4.0/cdi-tck-4.0.5-dist.zip >>> >>> [9] >>> https://download.eclipse.org/jakartaee/bean-validation/3.0/beanvalidation-tck-dist-3.0.1.zip >>> >>> >>> [10] >>> https://download.eclipse.org/jakartaee/dependency-injection/2.0/jakarta.inject-tck-2.0.2-bin.zip >>> >>> >>> [11] >>> https://download.eclipse.org/jakartaee/debugging/2.0/jakarta-debugging-tck-2.0.0.zip >>> >>> >>> [12] https://tomee.apache.org/9.0.0-M7/plume/webprofile-9.1.html >>> >>> [13] >>> https://jakarta.ee/specifications/webprofile/10/jakarta-webprofile-spec-10.0#a43 >>> >>> >>> >>> >>> >>> >>> ---- Em qui, 25 jul 2024 02:38:32 -0300 Richard Zowalla >>> <mailto:r...@apache.org> escreveu --- >>> >>> >>> >>> Hi David, >>> >>> Thanks for clarification! >>> >>> Gruß >>> Richard >>> >>>> Am 25.07.2024 um 00:39 schrieb David Blevins >>>> <mailto:mailto:david.blev...@gmail.com>: >>>> >>>> Hey All, >>>> >>>> We do need to still run and pass the Web Profile tests of the Platform >>>> TCK. More tests have been broken out in the same fashion CDI and >>>> Validation were always broken out, but there are still a great deal of >>>> tests that are not broken out and unlikely to ever be -- connector, ejb >>>> and certain integration tests in particular. >>>> >>>> Here is a certification result that includes tests from the Platform TCK >>>> and the name, version and SHA of the TCK binary used: >>>> >>>> - >>>> https://glassfish.org/certifications/jakarta-platform/10/TCK-Results-7.0.0-M8 >>>> >>>> >>>> Those can be found from this page: >>>> >>>> - https://jakarta.ee/compatibility/certification/10/ >>>> >>>> >>>> -David >>>> >>>>> On Jul 23, 2024, at 12:05 PM, Richard Zowalla >>>>> <mailto:mailto:r...@apache.org> wrote: >>>>> >>>>> Hey guys, >>>>> >>>>> do we actually need to run the platform-tck? >>>>> >>>>> Markus and myself were looking into it and for example Payara and Wildly >>>>> are doing standalone setups only: >>>>> >>>>> - https://github.com/payara/jakartaee-10-tck-runners >>>>> - https://github.com/wildfly/wildfly-tck-runners/tree/wildfly-27 >>>>> >>>>> So do we actually need the javatest based setup at all(aka platform-tck) >>>>> ?? >>>>> >>>>> Or can we also just do a bunch of standalone tck setups (as we already to >>>>> in /tck in the TomEE repo) and claim to be compliant if we pass the set >>>>> needed for the web profile? >>>>> >>>>> Hoping for someone, who is more familiar with the certification process >>>>> (David, maybe??) to jump in to give some guidance? >>>>> >>>>> Gruß >>>>> Richard >>>>> >>>>> >>>>>> Am 23.07.2024 um 12:53 schrieb Jonathan Gallimore >>>>>> <mailto:mailto:jonathan.gallim...@gmail.com>: >>>>>> >>>>>>> Also I'm far away from doing a complete run, I only ran a couple dozen >>>>>>> tests so far. Ideally once we figure out these setup issues someone >>>>>>> from >>>>>>> tomitribe could trigger a full run on tck.work so we can get some >>>>>>> actual >>>>>>> numbers. >>>>>> >>>>>> I'm happy to help wire that up. >>>>>> >>>>>> Jon >>>>>> >>>>>> On Tue, Jul 23, 2024 at 11:32 AM Markus Jung >>>>>> <mailto:mailto:ju...@apache.org> wrote: >>>>>> >>>>>>> Hey Richard, >>>>>>> >>>>>>> >>>>>>> I did some initial work on running a few TCK tests in the platform-tck >>>>>>> the past days and have been able to gain some quick wins with a couple >>>>>>> of setup issues. [1] >>>>>>> >>>>>>> >>>>>>> This is where I'm at: >>>>>>> Main pain points are that LegacyCookieProcessor is gone in Tomcat 10.1 >>>>>>> which causes a bunch of tests to fail and tssv.jar is seemingly >>>>>>> missing >>>>>>> from the platform-tck, which makes the ProviderConfiguration [2] >>>>>>> invalid >>>>>>> as this .jar contains the class >>>>>>> com.sun.ts.tests.jaspic.tssv.config.TSAuthConfigProvider. Updating the >>>>>>> Context.xml configs to not reference LegacyCookieProcesor anymore did >>>>>>> the trick but I'm still trying to figure out what needs to be done to >>>>>>> get the tssv.jar file. It was in the EE 9.1 TCK but is seemingly >>>>>>> missing >>>>>>> from EE 10. One solution that works is just extracting it from the >>>>>>> Authentication 3.0 TCK, but I doubt this is the right solution. Maybe >>>>>>> someone who is more involved knows more? >>>>>>> >>>>>>> Also I'm far away from doing a complete run, I only ran a couple dozen >>>>>>> tests so far. Ideally once we figure out these setup issues someone >>>>>>> from >>>>>>> tomitribe could trigger a full run on tck.work so we can get some >>>>>>> actual >>>>>>> numbers. >>>>>>> >>>>>>> Thanks >>>>>>> Markus >>>>>>> >>>>>>> [1] https://github.com/jungm/tomee-tck/tree/ee10work >>>>>>> [2] >>>>>>> >>>>>>> https://github.com/jungm/tomee-tck/blob/main/src/test/tomee-plume/conf/ProviderConfiguration.xml >>>>>>> >>>>>>> >>>>>>> On 23.07.24 11:06, Richard Zowalla wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> TomEE 10.0.0-M2 is out and it seems, that we have quite a few people >>>>>>> already testing it, which is great. >>>>>>>> >>>>>>>> Since we resolved most of the open dependency updates related to EE10, >>>>>>> we can now start to work on TCK setup / compliance or at least to get >>>>>>> some >>>>>>> numbers. >>>>>>>> Aside from the standalone TCKs here [1], we will need to check on how >>>>>>>> to >>>>>>> run https://github.com/jakartaee/platform-tck/tree/10.0.x with TomEE 10 >>>>>>> to (at least) get some numbers. >>>>>>>> Some of the required tweaks / config options (etc) can be found here: >>>>>>> https://github.com/apache/tomee-tck >>>>>>>> >>>>>>>> I think, that Markus J. did a quick check on the platform-tck, so maybe >>>>>>> you can add your findings in this thread, so we can (hopefully) create >>>>>>> an >>>>>>> environment for further discussion on how to proceed with the TCK >>>>>>> itself. >>>>>>>> >>>>>>>> From what I remember on the standalone TCKs: >>>>>>>> >>>>>>>> - We need to correctly setup the concurrency TCK and fix our >>>>>>> implementation if needed. >>>>>>>> - We need to check on the CDI TCK. >>>>>>>> >>>>>>>> All TCKs can be enabled via a specific Maven profile. >>>>>>>> >>>>>>>> There are still some other open things: >>>>>>>> >>>>>>>> - MP 6.0 requires us to implement OpenTelemetry, which hangs because of >>>>>>> https://issues.apache.org/jira/browse/OWB-1441 atm. >>>>>>>> - CXF 4.1.0 needs to be final ;-) >>>>>>>> >>>>>>>> So if anyone wants to have a look into one of these areas, feel free to >>>>>>> drop a message on dev@ >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Richard >>>>>>>> >>>>>>>> >>>>>>>> [1] https://github.com/apache/tomee/tree/main/tck >>>>>>> >>>>> >
