Hello, I write to summarize the state of things regarding Jenkins CI and ask for active involvement from the developers of each of the relevant projects - drools, kogito, optaplanner.
The most pressing topics are currently nightly builds. For several reasons we have unstable or failing builds. I'll try to summarize below. 0. Drools nightlies are green in all default, native and native-lts environments, and should be in fairly good shape for branching ( https://ci-builds.apache.org/job/KIE/job/drools/job/main/). 1. Nightly build timeouts for kogito-apps and kogito-examples - this turns out to be a regular issue, so I am gonna increase the timeouts today. 2. Unstable tests in kogito-runtimes, kogito-apps and kogito-examples - generally there still seem to be unstable tests in these repositories mainly, it would be beneficial instead of repeated rerunning and "merging while it's green" by chance at least track and investigate the failure, ideally providing a fix. 3. Nightly build for kogito-runtimes in native mode fails due to Python libraries issue, which does not appear in default environment (jvm mode) so I suspect the failing module might not be well configured for native mode? This needs an SME to resolve. ( https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-runtimes.build-and-test/ ) 4. Nightly build for kogito-apps in native mode, numerous data-index IT failures, I have not seen those in PR checks or default environment in nightly, so again I suspect this might be due to native mode itself and need help investigating. ( https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-apps.build-and-test/19/testReport/ ) 5. Optaplanner PR checks and nightlies are slightly behind on fixing, as I haven't seen any urgent demand on fixing those as a priority, so I focused on areas with active development. Yesterday we enabled nightly triggers, currently the builds are failing due to pipeline misconfig, that's something I can take a look at. But SMEs should actually review the state of things, I haven't heard from anyone. 6. Images PR checks and pipelines - these are also falling behind, and unless someone with detailed knowledge joins the efforts it might be a lengthy process. I consider this being the next priority like we've agreed on that in the last sync meeting. I should make some progress during this week on the task. 7. SonarCloud - we're still not onboarded in apache organization because of still existing projects under kiegroup - which results in conflict during import. This is being looked at and hopefully we have the grounds to import projects under the apache organization soon. 8. Apache Nexus onboarding - as Tibor mentioned in https://lists.apache.org/thread/rk718nbww2jr3h4wqv6fbm656vo5k998 , hopefully soon we have some progress in onboarding. From the view of pipelines this should be configuring correct credentials and enabling nightly deployment in the configuration. And then we'll see where we're at. 9. For branching itself, the pipelines have not yet been validated, but at least now we have a token with push privileges, which blocked us until now. 10. As we're not yet discussing release, I'll touch on that just briefly, I personally haven't yet looked past nightlies, so it's hard to assess the relevance or status of it in the new environment. In the end, I'll summarize the job locations, so that anyone can step in and help where's needed: https://ci-builds.apache.org/job/KIE/job/drools/ https://ci-builds.apache.org/job/KIE/job/kogito/ https://ci-builds.apache.org/job/KIE/job/optaplanner/ I am here to share the context of course. Regards Jan On Mon, 9 Oct 2023 at 13:26, Jan Šťastný <[email protected]> wrote: > To update on the PR checks failures: > - Installation of the missing libraries seems to have done the trick. > > Now most of the test failures that are left are seemingly random fails > during Keycloak container startup. I suspect the reason is that the > startTimeout which is defined in the base class for all kogito > containers is overridden by mistake using a consecutive call to waitFor > (both being testcontainers API specifics). Anyway tracked as > https://github.com/apache/incubator-kie-issues/issues/622 for more > insights (and PR at > https://github.com/apache/incubator-kie-kogito-runtimes/pull/3247 ). > > That should limit the number of failing tests and facilitate > identification of other failures, so hopefully will be fixed and merged > soon. > > Regards > Jan > > On Thu, 5 Oct 2023 at 18:45, Jan Šťastný <[email protected]> wrote: > >> There's a missing system library as noted in the issue, I'll start with >> that and reach out if it doesn't resolve the issue. >> >> Dne čt 5. 10. 2023 18:28 uživatel Francisco Javier Tirado Sarti < >> [email protected]> napsal: >> >>> The openvino one is related to the way python dependencies are resolved. >>> Somewhere in the CI machine JEP dependency is being installed together >>> with numpy, but this causes JEP to not include numpy in the wheel, because >>> both are invoked by the same PIP command (this is a guess) >>> Basically, we need to make sure that pip is not invoked or invoked >>> twice (as in the maven file for that example >>> https://github.com/apache/incubator-kie-kogito-examples/blob/stable/serverless-workflow-examples/serverless-workflow-openvino-quarkus/pom.xml#L138-L161), >>> first with numpy and later with JEP and other dependencies. >>> Hope this helps >>> >>> On Thu, Oct 5, 2023 at 6:08 PM Jan Šťastný <[email protected]> >>> wrote: >>> >>>> I've noticed a few outstanding problems with the jenkins environment, >>>> tracking under >>>> https://github.com/apache/incubator-kie-issues/issues/604 , >>>> will fix that tomorrow. It affects only a few tests in >>>> incubator-kie-kogito-examples: >>>> >>>> org.kie.kogito.quarkus.outbox.OutboxIT.(?) >>>> org.kie.kogito.springboot.outbox.OutboxIT.(?) >>>> org.kie.kogito.examples.OpenVinoRestIT.testOpenvino >>>> >>>> Regards >>>> Jan >>>> >>>> On Thu, 5 Oct 2023 at 14:14, Jan Šťastný <[email protected]> wrote: >>>> >>>> > Thanks to Roberto's efforts, the build-chain is fixed and released. >>>> Many >>>> > thanks. >>>> > >>>> > Taking Enrique's multi-repository PR as an example, build-chain >>>> finally >>>> > reports similar: >>>> > --- >>>> > # Checkout summary >>>> > [INFO] [apache/incubator-kie-drools] >>>> > [INFO] Project taken from apache/incubator-kie-drools:main (Branch >>>> HEAD >>>> > 1dda51ba5d4c60e293780fcbbdf736a1f52a4031) >>>> > [INFO] Merged elguardian/drools:kie-issues-249 into branch main >>>> > [INFO] [apache/incubator-kie-kogito-runtimes] >>>> > [INFO] Project taken from apache/incubator-kie-kogito-runtimes:main >>>> > (Branch HEAD 437a3167560b5260ec949f9e126967a91a00155d) >>>> > [INFO] Merged elguardian/kogito-runtimes:kie-issues-249 into branch >>>> main >>>> > [INFO] [apache/incubator-kie-kogito-apps] >>>> > [INFO] Project taken from apache/incubator-kie-kogito-apps:main >>>> (Branch >>>> > HEAD 73c27d909b577c2a21782deef91e68cbc2301b85) >>>> > [INFO] Merged elguardian/kogito-apps:kie-issues-249 into branch main >>>> > [INFO] [apache/incubator-kie-kogito-examples] >>>> > [INFO] Project taken from apache/incubator-kie-kogito-examples:main >>>> > (Branch HEAD 8663ef433056a278ae2657258f0ab366edc9abc5) >>>> > [INFO] Merged elguardian/kogito-examples:kie-issues-249 into branch >>>> main >>>> > --- >>>> > >>>> > It does require retriggering the PR checks, which I have started to >>>> do. >>>> > >>>> > There are numerous cases of old/stale PRs that need to be rebased or >>>> > closed though. Please review your projects. >>>> > >>>> > Regards >>>> > Jan >>>> > >>>> > On Thu, 5 Oct 2023 at 12:38, Francisco Javier Tirado Sarti < >>>> > [email protected]> wrote: >>>> > >>>> >> I forgot to copy past the link of the PR >>>> >> https://github.com/apache/incubator-kie-kogito-runtimes/pull/3227 >>>> >> >>>> >> On Thu, Oct 5, 2023 at 11:00 AM Francisco Javier Tirado Sarti < >>>> >> [email protected]> wrote: >>>> >> >>>> >> > Hi Jan, >>>> >> > I just rebased this one (one of the 5 I created pending on >>>> runtimes), >>>> >> but >>>> >> > I cannot see the jenkins pipeline being executed (the gha failing >>>> is >>>> >> > expected) >>>> >> > Am I missing something? >>>> >> > >>>> >> > On Thu, Oct 5, 2023 at 10:55 AM Jan Šťastný <[email protected] >>>> > >>>> >> wrote: >>>> >> > >>>> >> >> Thanks for bringing that up. >>>> >> >> >>>> >> >> You're obviously right, I didn't account for the fact that the >>>> >> build-chain >>>> >> >> tool handles all on its own.We're blocked until the PR above is >>>> merged >>>> >> and >>>> >> >> a new release in NPM appears. Given that it would be a bugfix I >>>> suspect >>>> >> >> existing version specifier would take the latest available: >>>> >> >> >>>> >> >> kiegroup/github-action-build-chain@v3 >>>> >> >> >>>> >> >> which is derived using >>>> >> >> >>>> >> >> >>>> >> >>>> https://github.com/kiegroup/jenkins-pipeline-shared-libraries/blob/main/vars/buildChain.groovy >>>> >> >> from >>>> >> >> >>>> >> >> >>>> >> >>>> https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/actions/build-chain/action.yml#L36 >>>> >> >> . >>>> >> >> >>>> >> >> Sorry for the mistake in summary and guidance. So merging is not >>>> safe >>>> >> >> judging on the PR checks results until we switch to updated >>>> >> build-chain. >>>> >> >> >>>> >> >> Regards >>>> >> >> Jan >>>> >> >> >>>> >> >> On Thu, 5 Oct 2023 at 10:45, Marek Novotny <[email protected]> >>>> >> wrote: >>>> >> >> >>>> >> >> > yes, that is correct Toshiya. For some reason the different >>>> named >>>> >> forked >>>> >> >> > repository is not correctly check out by Build chain used in CI >>>> job. >>>> >> >> > >>>> >> >> > The PR >>>> >> https://github.com/kiegroup/github-action-build-chain/pull/462 >>>> >> >> > should fix that and then all CI jobs need to upgrade to new >>>> version >>>> >> of >>>> >> >> BC >>>> >> >> > to run with that fixed version of BC. >>>> >> >> > >>>> >> >> > BR, >>>> >> >> > Dne 10/5/23 v 09:59 Toshiya Kobayashi napsal(a): >>>> >> >> > >>>> >> >> > Thank you very much for the progress, Jan and the people >>>> involved! >>>> >> >> > >>>> >> >> > One question: >>>> >> >> > >>>> >> >> > >>>> >> >> > We've already seen green PR checks in incubator-kie-drools >>>> >> >> > >>>> >> >> > I raised a PR and checked the Jenkins CI yesterday. >>>> >> >> > https://github.com/apache/incubator-kie-drools/pull/5545 >>>> >> >> > >>>> >> >> > ``` >>>> >> >> > # Checkout summary >>>> >> >> > [INFO] [apache/incubator-kie-drools] >>>> >> >> > [INFO] Project taken from apache/incubator-kie-drools:main >>>> (Branch >>>> >> >> HEAD >>>> >> >> > 02d592c720e43ca028d56e22375a750a9097f694) >>>> >> >> > [INFO] [apache/incubator-kie-kogito-runtimes] >>>> >> >> > [INFO] Project taken from >>>> apache/incubator-kie-kogito-runtimes:main >>>> >> >> > (Branch HEAD 2103ef90832274fb2aeeb55b6e5180263d2ea017) >>>> >> >> > [INFO] [apache/incubator-kie-kogito-apps] >>>> >> >> > [INFO] Project taken from >>>> apache/incubator-kie-kogito-apps:main >>>> >> >> (Branch >>>> >> >> > HEAD 507ede90bd186f3f6e7fba8454faf19967acfc1f) >>>> >> >> > [INFO] [apache/incubator-kie-kogito-examples] >>>> >> >> > [INFO] Project taken from >>>> apache/incubator-kie-kogito-examples:main >>>> >> >> > (Branch HEAD 868814b52c2ea2caadb839f4652a4cc26fe7912a) >>>> >> >> > ``` >>>> >> >> > >>>> >> >> > It didn't pick my PR branch >>>> >> `tkobayas/drools:DROOLS-7570-test-hang-2`. >>>> >> >> (my >>>> >> >> > fork repo name is `drools`, not `incubator-kie-drools`) >>>> >> >> > >>>> >> >> > So don't we need to wait forhttps:// >>>> >> >> github.com/kiegroup/github-action-build-chain/pull/462 even for a >>>> >> >> > single repo PR? >>>> >> >> > >>>> >> >> > Thanks! >>>> >> >> > Toshiya >>>> >> >> > >>>> >> >> > On Thu, Oct 5, 2023 at 3:39 AM Jan Šťastný < >>>> [email protected]> < >>>> >> >> [email protected]> wrote: >>>> >> >> > >>>> >> >> > >>>> >> >> > Hello, >>>> >> >> > we've managed to progress in some areas. Small steps. >>>> >> >> > >>>> >> >> > [PRs] >>>> >> >> > We've already seen green PR checks in incubator-kie-drools and >>>> >> >> > incubator-kie-kogito-runtimes repositories. If you have a >>>> pending PR, >>>> >> >> make >>>> >> >> > sure you rebase with the latest changes from main, PR checks' >>>> >> behavior >>>> >> >> > depends on that. >>>> >> >> > >>>> >> >> > [Nightly] >>>> >> >> > Recent problems with triggers resolved, tonight should trigger >>>> first >>>> >> >> drools >>>> >> >> > nightly, to confirm the setup, other repos will be enabled >>>> after. >>>> >> >> > >>>> >> >> > With this we've already found several issues, for some we don't >>>> have >>>> >> >> > workaround yet. >>>> >> >> > >>>> >> >> > Most problematic is our inability to run multi-repository >>>> builds for >>>> >> >> PRs, >>>> >> >> > so if we have changes related to each other in 3 related >>>> repositories >>>> >> >> they >>>> >> >> > are not at the moment tested together. Which is a big problem >>>> that is >>>> >> >> being >>>> >> >> > tackled already inhttps:// >>>> >> >> github.com/kiegroup/github-action-build-chain/pull/462 . >>>> >> >> > >>>> >> >> > We're also seeing docker-related issues in execution of >>>> >> >> > incubator-kie-kogito-apps and incubator-kie-kogito-examples PR >>>> >> checks, >>>> >> >> in >>>> >> >> > most cases due to keycloak containers not being started - anyone >>>> >> >> willing to >>>> >> >> > take a look, please jump on that, ping me if needed. >>>> >> >> > >>>> >> >> > We've also turned everything possible off, and we need to >>>> gradually >>>> >> put >>>> >> >> > things back to shape. For immediate tasks I would name Apache >>>> Nexus >>>> >> >> > snapshot deployment, SonarCloud reporting and slightly related - >>>> >> >> > fine-grained PR checks. >>>> >> >> > >>>> >> >> > Regards >>>> >> >> > Jan >>>> >> >> > >>>> >> >> > >>>> >> >> > -- >>>> >> >> > Marek Novotny >>>> >> >> > -- >>>> >> >> > >>>> >> >> > RedHat JBoss Middleware >>>> >> >> > >>>> >> >> > Red Hat Czech s.r.o. >>>> >> >> > Purkynova 111 >>>> >> >> > 612 45 Brno >>>> >> >> > >>>> >> >> > >>>> >> >> >>>> >> > >>>> >> >>>> > >>>> >>>
