If you are interested, you can follow up my history with jep and native here https://github.com/oracle/graal/issues/6345
On 2023/10/24 16:58:48 Francisco Javier Tirado Sarti wrote: > Hi, > Regarding 3), I fixed locally by setting LD_LIBRARY_PATH env variable to > GRAALVM_HOME/lib/server. > It is possible to do the same in the Jenkins virtual machine? > > > On 2023/10/24 12:24:53 Jan Šťastný wrote: > > Hello, > > > > I write to summarize the state of things regarding Jenkins CI and ask for > > active involvement from the developers of each of the relevant projects - > > drools, kogito, optaplanner. > > > > The most pressing topics are currently nightly builds. > > > > For several reasons we have unstable or failing builds. I'll try to > > summarize below. > > > > 0. Drools nightlies are green in all default, native and native-lts > > environments, and should be in fairly good shape for branching ( > > https://ci-builds.apache.org/job/KIE/job/drools/job/main/). > > 1. Nightly build timeouts for kogito-apps and kogito-examples - this turns > > out to be a regular issue, so I am gonna increase the timeouts today. > > 2. Unstable tests in kogito-runtimes, kogito-apps and kogito-examples - > > generally there still seem to be unstable tests in these repositories > > mainly, it would be beneficial instead of repeated rerunning and "merging > > while it's green" by chance at least track and investigate the failure, > > ideally providing a fix. > > 3. Nightly build for kogito-runtimes in native mode fails due to Python > > libraries issue, which does not appear in default environment (jvm mode) so > > I suspect the failing module might not be well configured for native mode? > > This needs an SME to resolve. ( > > https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-runtimes.build-and-test/ > > ) > > 4. Nightly build for kogito-apps in native mode, numerous data-index IT > > failures, I have not seen those in PR checks or default environment in > > nightly, so again I suspect this might be due to native mode itself and > > need help investigating. ( > > https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-apps.build-and-test/19/testReport/ > > ) > > 5. Optaplanner PR checks and nightlies are slightly behind on fixing, as I > > haven't seen any urgent demand on fixing those as a priority, so I focused > > on areas with active development. Yesterday we enabled nightly triggers, > > currently the builds are failing due to pipeline misconfig, that's > > something I can take a look at. But SMEs should actually review the state > > of things, I haven't heard from anyone. > > 6. Images PR checks and pipelines - these are also falling behind, and > > unless someone with detailed knowledge joins the efforts it might be a > > lengthy process. I consider this being the next priority like we've agreed > > on that in the last sync meeting. I should make some progress during this > > week on the task. > > 7. SonarCloud - we're still not onboarded in apache organization because of > > still existing projects under kiegroup - which results in conflict during > > import. This is being looked at and hopefully we have the grounds to import > > projects under the apache organization soon. > > 8. Apache Nexus onboarding - as Tibor mentioned in > > https://lists.apache.org/thread/rk718nbww2jr3h4wqv6fbm656vo5k998 , > > hopefully soon we have some progress in onboarding. From the view of > > pipelines this should be configuring correct credentials and enabling > > nightly deployment in the configuration. And then we'll see where we're at. > > 9. For branching itself, the pipelines have not yet been validated, but at > > least now we have a token with push privileges, which blocked us until now. > > 10. As we're not yet discussing release, I'll touch on that just briefly, I > > personally haven't yet looked past nightlies, so it's hard to assess the > > relevance or status of it in the new environment. > > > > In the end, I'll summarize the job locations, so that anyone can step in > > and help where's needed: > > https://ci-builds.apache.org/job/KIE/job/drools/ > > https://ci-builds.apache.org/job/KIE/job/kogito/ > > https://ci-builds.apache.org/job/KIE/job/optaplanner/ > > > > I am here to share the context of course. > > > > Regards > > Jan > > > > On Mon, 9 Oct 2023 at 13:26, Jan Šťastný <[email protected]> wrote: > > > > > To update on the PR checks failures: > > > - Installation of the missing libraries seems to have done the trick. > > > > > > Now most of the test failures that are left are seemingly random fails > > > during Keycloak container startup. I suspect the reason is that the > > > startTimeout which is defined in the base class for all kogito > > > containers is overridden by mistake using a consecutive call to waitFor > > > (both being testcontainers API specifics). Anyway tracked as > > > https://github.com/apache/incubator-kie-issues/issues/622 for more > > > insights (and PR at > > > https://github.com/apache/incubator-kie-kogito-runtimes/pull/3247 ). > > > > > > That should limit the number of failing tests and facilitate > > > identification of other failures, so hopefully will be fixed and merged > > > soon. > > > > > > Regards > > > Jan > > > > > > On Thu, 5 Oct 2023 at 18:45, Jan Šťastný <[email protected]> wrote: > > > > > >> There's a missing system library as noted in the issue, I'll start with > > >> that and reach out if it doesn't resolve the issue. > > >> > > >> Dne čt 5. 10. 2023 18:28 uživatel Francisco Javier Tirado Sarti < > > >> [email protected]> napsal: > > >> > > >>> The openvino one is related to the way python dependencies are resolved. > > >>> Somewhere in the CI machine JEP dependency is being installed together > > >>> with numpy, but this causes JEP to not include numpy in the wheel, > > >>> because > > >>> both are invoked by the same PIP command (this is a guess) > > >>> Basically, we need to make sure that pip is not invoked or invoked > > >>> twice (as in the maven file for that example > > >>> https://github.com/apache/incubator-kie-kogito-examples/blob/stable/serverless-workflow-examples/serverless-workflow-openvino-quarkus/pom.xml#L138-L161), > > >>> first with numpy and later with JEP and other dependencies. > > >>> Hope this helps > > >>> > > >>> On Thu, Oct 5, 2023 at 6:08 PM Jan Šťastný <[email protected]> > > >>> wrote: > > >>> > > >>>> I've noticed a few outstanding problems with the jenkins environment, > > >>>> tracking under > > >>>> https://github.com/apache/incubator-kie-issues/issues/604 , > > >>>> will fix that tomorrow. It affects only a few tests in > > >>>> incubator-kie-kogito-examples: > > >>>> > > >>>> org.kie.kogito.quarkus.outbox.OutboxIT.(?) > > >>>> org.kie.kogito.springboot.outbox.OutboxIT.(?) > > >>>> org.kie.kogito.examples.OpenVinoRestIT.testOpenvino > > >>>> > > >>>> Regards > > >>>> Jan > > >>>> > > >>>> On Thu, 5 Oct 2023 at 14:14, Jan Šťastný <[email protected]> wrote: > > >>>> > > >>>> > Thanks to Roberto's efforts, the build-chain is fixed and released. > > >>>> Many > > >>>> > thanks. > > >>>> > > > >>>> > Taking Enrique's multi-repository PR as an example, build-chain > > >>>> finally > > >>>> > reports similar: > > >>>> > --- > > >>>> > # Checkout summary > > >>>> > [INFO] [apache/incubator-kie-drools] > > >>>> > [INFO] Project taken from apache/incubator-kie-drools:main (Branch > > >>>> HEAD > > >>>> > 1dda51ba5d4c60e293780fcbbdf736a1f52a4031) > > >>>> > [INFO] Merged elguardian/drools:kie-issues-249 into branch main > > >>>> > [INFO] [apache/incubator-kie-kogito-runtimes] > > >>>> > [INFO] Project taken from apache/incubator-kie-kogito-runtimes:main > > >>>> > (Branch HEAD 437a3167560b5260ec949f9e126967a91a00155d) > > >>>> > [INFO] Merged elguardian/kogito-runtimes:kie-issues-249 into branch > > >>>> main > > >>>> > [INFO] [apache/incubator-kie-kogito-apps] > > >>>> > [INFO] Project taken from apache/incubator-kie-kogito-apps:main > > >>>> (Branch > > >>>> > HEAD 73c27d909b577c2a21782deef91e68cbc2301b85) > > >>>> > [INFO] Merged elguardian/kogito-apps:kie-issues-249 into branch main > > >>>> > [INFO] [apache/incubator-kie-kogito-examples] > > >>>> > [INFO] Project taken from apache/incubator-kie-kogito-examples:main > > >>>> > (Branch HEAD 8663ef433056a278ae2657258f0ab366edc9abc5) > > >>>> > [INFO] Merged elguardian/kogito-examples:kie-issues-249 into branch > > >>>> main > > >>>> > --- > > >>>> > > > >>>> > It does require retriggering the PR checks, which I have started to > > >>>> do. > > >>>> > > > >>>> > There are numerous cases of old/stale PRs that need to be rebased or > > >>>> > closed though. Please review your projects. > > >>>> > > > >>>> > Regards > > >>>> > Jan > > >>>> > > > >>>> > On Thu, 5 Oct 2023 at 12:38, Francisco Javier Tirado Sarti < > > >>>> > [email protected]> wrote: > > >>>> > > > >>>> >> I forgot to copy past the link of the PR > > >>>> >> https://github.com/apache/incubator-kie-kogito-runtimes/pull/3227 > > >>>> >> > > >>>> >> On Thu, Oct 5, 2023 at 11:00 AM Francisco Javier Tirado Sarti < > > >>>> >> [email protected]> wrote: > > >>>> >> > > >>>> >> > Hi Jan, > > >>>> >> > I just rebased this one (one of the 5 I created pending on > > >>>> runtimes), > > >>>> >> but > > >>>> >> > I cannot see the jenkins pipeline being executed (the gha failing > > >>>> is > > >>>> >> > expected) > > >>>> >> > Am I missing something? > > >>>> >> > > > >>>> >> > On Thu, Oct 5, 2023 at 10:55 AM Jan Šťastný <[email protected] > > >>>> > > > >>>> >> wrote: > > >>>> >> > > > >>>> >> >> Thanks for bringing that up. > > >>>> >> >> > > >>>> >> >> You're obviously right, I didn't account for the fact that the > > >>>> >> build-chain > > >>>> >> >> tool handles all on its own.We're blocked until the PR above is > > >>>> merged > > >>>> >> and > > >>>> >> >> a new release in NPM appears. Given that it would be a bugfix I > > >>>> suspect > > >>>> >> >> existing version specifier would take the latest available: > > >>>> >> >> > > >>>> >> >> kiegroup/github-action-build-chain@v3 > > >>>> >> >> > > >>>> >> >> which is derived using > > >>>> >> >> > > >>>> >> >> > > >>>> >> > > >>>> https://github.com/kiegroup/jenkins-pipeline-shared-libraries/blob/main/vars/buildChain.groovy > > >>>> >> >> from > > >>>> >> >> > > >>>> >> >> > > >>>> >> > > >>>> https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/actions/build-chain/action.yml#L36 > > >>>> >> >> . > > >>>> >> >> > > >>>> >> >> Sorry for the mistake in summary and guidance. So merging is not > > >>>> safe > > >>>> >> >> judging on the PR checks results until we switch to updated > > >>>> >> build-chain. > > >>>> >> >> > > >>>> >> >> Regards > > >>>> >> >> Jan > > >>>> >> >> > > >>>> >> >> On Thu, 5 Oct 2023 at 10:45, Marek Novotny <[email protected]> > > >>>> >> wrote: > > >>>> >> >> > > >>>> >> >> > yes, that is correct Toshiya. For some reason the different > > >>>> named > > >>>> >> forked > > >>>> >> >> > repository is not correctly check out by Build chain used in CI > > >>>> job. > > >>>> >> >> > > > >>>> >> >> > The PR > > >>>> >> https://github.com/kiegroup/github-action-build-chain/pull/462 > > >>>> >> >> > should fix that and then all CI jobs need to upgrade to new > > >>>> version > > >>>> >> of > > >>>> >> >> BC > > >>>> >> >> > to run with that fixed version of BC. > > >>>> >> >> > > > >>>> >> >> > BR, > > >>>> >> >> > Dne 10/5/23 v 09:59 Toshiya Kobayashi napsal(a): > > >>>> >> >> > > > >>>> >> >> > Thank you very much for the progress, Jan and the people > > >>>> involved! > > >>>> >> >> > > > >>>> >> >> > One question: > > >>>> >> >> > > > >>>> >> >> > > > >>>> >> >> > We've already seen green PR checks in incubator-kie-drools > > >>>> >> >> > > > >>>> >> >> > I raised a PR and checked the Jenkins CI yesterday. > > >>>> >> >> > https://github.com/apache/incubator-kie-drools/pull/5545 > > >>>> >> >> > > > >>>> >> >> > ``` > > >>>> >> >> > # Checkout summary > > >>>> >> >> > [INFO] [apache/incubator-kie-drools] > > >>>> >> >> > [INFO] Project taken from apache/incubator-kie-drools:main > > >>>> (Branch > > >>>> >> >> HEAD > > >>>> >> >> > 02d592c720e43ca028d56e22375a750a9097f694) > > >>>> >> >> > [INFO] [apache/incubator-kie-kogito-runtimes] > > >>>> >> >> > [INFO] Project taken from > > >>>> apache/incubator-kie-kogito-runtimes:main > > >>>> >> >> > (Branch HEAD 2103ef90832274fb2aeeb55b6e5180263d2ea017) > > >>>> >> >> > [INFO] [apache/incubator-kie-kogito-apps] > > >>>> >> >> > [INFO] Project taken from > > >>>> apache/incubator-kie-kogito-apps:main > > >>>> >> >> (Branch > > >>>> >> >> > HEAD 507ede90bd186f3f6e7fba8454faf19967acfc1f) > > >>>> >> >> > [INFO] [apache/incubator-kie-kogito-examples] > > >>>> >> >> > [INFO] Project taken from > > >>>> apache/incubator-kie-kogito-examples:main > > >>>> >> >> > (Branch HEAD 868814b52c2ea2caadb839f4652a4cc26fe7912a) > > >>>> >> >> > ``` > > >>>> >> >> > > > >>>> >> >> > It didn't pick my PR branch > > >>>> >> `tkobayas/drools:DROOLS-7570-test-hang-2`. > > >>>> >> >> (my > > >>>> >> >> > fork repo name is `drools`, not `incubator-kie-drools`) > > >>>> >> >> > > > >>>> >> >> > So don't we need to wait forhttps:// > > >>>> >> >> github.com/kiegroup/github-action-build-chain/pull/462 even for a > > >>>> >> >> > single repo PR? > > >>>> >> >> > > > >>>> >> >> > Thanks! > > >>>> >> >> > Toshiya > > >>>> >> >> > > > >>>> >> >> > On Thu, Oct 5, 2023 at 3:39 AM Jan Šťastný < > > >>>> [email protected]> < > > >>>> >> >> [email protected]> wrote: > > >>>> >> >> > > > >>>> >> >> > > > >>>> >> >> > Hello, > > >>>> >> >> > we've managed to progress in some areas. Small steps. > > >>>> >> >> > > > >>>> >> >> > [PRs] > > >>>> >> >> > We've already seen green PR checks in incubator-kie-drools and > > >>>> >> >> > incubator-kie-kogito-runtimes repositories. If you have a > > >>>> pending PR, > > >>>> >> >> make > > >>>> >> >> > sure you rebase with the latest changes from main, PR checks' > > >>>> >> behavior > > >>>> >> >> > depends on that. > > >>>> >> >> > > > >>>> >> >> > [Nightly] > > >>>> >> >> > Recent problems with triggers resolved, tonight should trigger > > >>>> first > > >>>> >> >> drools > > >>>> >> >> > nightly, to confirm the setup, other repos will be enabled > > >>>> after. > > >>>> >> >> > > > >>>> >> >> > With this we've already found several issues, for some we don't > > >>>> have > > >>>> >> >> > workaround yet. > > >>>> >> >> > > > >>>> >> >> > Most problematic is our inability to run multi-repository > > >>>> builds for > > >>>> >> >> PRs, > > >>>> >> >> > so if we have changes related to each other in 3 related > > >>>> repositories > > >>>> >> >> they > > >>>> >> >> > are not at the moment tested together. Which is a big problem > > >>>> that is > > >>>> >> >> being > > >>>> >> >> > tackled already inhttps:// > > >>>> >> >> github.com/kiegroup/github-action-build-chain/pull/462 . > > >>>> >> >> > > > >>>> >> >> > We're also seeing docker-related issues in execution of > > >>>> >> >> > incubator-kie-kogito-apps and incubator-kie-kogito-examples PR > > >>>> >> checks, > > >>>> >> >> in > > >>>> >> >> > most cases due to keycloak containers not being started - > > >>>> >> >> > anyone > > >>>> >> >> willing to > > >>>> >> >> > take a look, please jump on that, ping me if needed. > > >>>> >> >> > > > >>>> >> >> > We've also turned everything possible off, and we need to > > >>>> gradually > > >>>> >> put > > >>>> >> >> > things back to shape. For immediate tasks I would name Apache > > >>>> Nexus > > >>>> >> >> > snapshot deployment, SonarCloud reporting and slightly related > > >>>> >> >> > - > > >>>> >> >> > fine-grained PR checks. > > >>>> >> >> > > > >>>> >> >> > Regards > > >>>> >> >> > Jan > > >>>> >> >> > > > >>>> >> >> > > > >>>> >> >> > -- > > >>>> >> >> > Marek Novotny > > >>>> >> >> > -- > > >>>> >> >> > > > >>>> >> >> > RedHat JBoss Middleware > > >>>> >> >> > > > >>>> >> >> > Red Hat Czech s.r.o. > > >>>> >> >> > Purkynova 111 > > >>>> >> >> > 612 45 Brno > > >>>> >> >> > > > >>>> >> >> > > > >>>> >> >> > > >>>> >> > > > >>>> >> > > >>>> > > > >>>> > > >>> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
