Im looking into point 4, got it locally reproduced On Tue, Oct 24, 2023 at 2:25 PM Jan Šťastný <[email protected]> wrote:
> Hello, > > I write to summarize the state of things regarding Jenkins CI and ask for > active involvement from the developers of each of the relevant projects - > drools, kogito, optaplanner. > > The most pressing topics are currently nightly builds. > > For several reasons we have unstable or failing builds. I'll try to > summarize below. > > 0. Drools nightlies are green in all default, native and native-lts > environments, and should be in fairly good shape for branching ( > https://ci-builds.apache.org/job/KIE/job/drools/job/main/). > 1. Nightly build timeouts for kogito-apps and kogito-examples - this turns > out to be a regular issue, so I am gonna increase the timeouts today. > 2. Unstable tests in kogito-runtimes, kogito-apps and kogito-examples - > generally there still seem to be unstable tests in these repositories > mainly, it would be beneficial instead of repeated rerunning and "merging > while it's green" by chance at least track and investigate the failure, > ideally providing a fix. > 3. Nightly build for kogito-runtimes in native mode fails due to Python > libraries issue, which does not appear in default environment (jvm mode) so > I suspect the failing module might not be well configured for native mode? > This needs an SME to resolve. ( > > https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-runtimes.build-and-test/ > ) > 4. Nightly build for kogito-apps in native mode, numerous data-index IT > failures, I have not seen those in PR checks or default environment in > nightly, so again I suspect this might be due to native mode itself and > need help investigating. ( > > https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-apps.build-and-test/19/testReport/ > ) > 5. Optaplanner PR checks and nightlies are slightly behind on fixing, as I > haven't seen any urgent demand on fixing those as a priority, so I focused > on areas with active development. Yesterday we enabled nightly triggers, > currently the builds are failing due to pipeline misconfig, that's > something I can take a look at. But SMEs should actually review the state > of things, I haven't heard from anyone. > 6. Images PR checks and pipelines - these are also falling behind, and > unless someone with detailed knowledge joins the efforts it might be a > lengthy process. I consider this being the next priority like we've agreed > on that in the last sync meeting. I should make some progress during this > week on the task. > 7. SonarCloud - we're still not onboarded in apache organization because of > still existing projects under kiegroup - which results in conflict during > import. This is being looked at and hopefully we have the grounds to import > projects under the apache organization soon. > 8. Apache Nexus onboarding - as Tibor mentioned in > https://lists.apache.org/thread/rk718nbww2jr3h4wqv6fbm656vo5k998 , > hopefully soon we have some progress in onboarding. From the view of > pipelines this should be configuring correct credentials and enabling > nightly deployment in the configuration. And then we'll see where we're at. > 9. For branching itself, the pipelines have not yet been validated, but at > least now we have a token with push privileges, which blocked us until now. > 10. As we're not yet discussing release, I'll touch on that just briefly, I > personally haven't yet looked past nightlies, so it's hard to assess the > relevance or status of it in the new environment. > > In the end, I'll summarize the job locations, so that anyone can step in > and help where's needed: > https://ci-builds.apache.org/job/KIE/job/drools/ > https://ci-builds.apache.org/job/KIE/job/kogito/ > https://ci-builds.apache.org/job/KIE/job/optaplanner/ > > I am here to share the context of course. > > Regards > Jan > > On Mon, 9 Oct 2023 at 13:26, Jan Šťastný <[email protected]> wrote: > > > To update on the PR checks failures: > > - Installation of the missing libraries seems to have done the trick. > > > > Now most of the test failures that are left are seemingly random fails > > during Keycloak container startup. I suspect the reason is that the > > startTimeout which is defined in the base class for all kogito > > containers is overridden by mistake using a consecutive call to waitFor > > (both being testcontainers API specifics). Anyway tracked as > > https://github.com/apache/incubator-kie-issues/issues/622 for more > > insights (and PR at > > https://github.com/apache/incubator-kie-kogito-runtimes/pull/3247 ). > > > > That should limit the number of failing tests and facilitate > > identification of other failures, so hopefully will be fixed and merged > > soon. > > > > Regards > > Jan > > > > On Thu, 5 Oct 2023 at 18:45, Jan Šťastný <[email protected]> wrote: > > > >> There's a missing system library as noted in the issue, I'll start with > >> that and reach out if it doesn't resolve the issue. > >> > >> Dne čt 5. 10. 2023 18:28 uživatel Francisco Javier Tirado Sarti < > >> [email protected]> napsal: > >> > >>> The openvino one is related to the way python dependencies are > resolved. > >>> Somewhere in the CI machine JEP dependency is being installed together > >>> with numpy, but this causes JEP to not include numpy in the wheel, > because > >>> both are invoked by the same PIP command (this is a guess) > >>> Basically, we need to make sure that pip is not invoked or invoked > >>> twice (as in the maven file for that example > >>> > https://github.com/apache/incubator-kie-kogito-examples/blob/stable/serverless-workflow-examples/serverless-workflow-openvino-quarkus/pom.xml#L138-L161 > ), > >>> first with numpy and later with JEP and other dependencies. > >>> Hope this helps > >>> > >>> On Thu, Oct 5, 2023 at 6:08 PM Jan Šťastný <[email protected]> > >>> wrote: > >>> > >>>> I've noticed a few outstanding problems with the jenkins environment, > >>>> tracking under > >>>> https://github.com/apache/incubator-kie-issues/issues/604 , > >>>> will fix that tomorrow. It affects only a few tests in > >>>> incubator-kie-kogito-examples: > >>>> > >>>> org.kie.kogito.quarkus.outbox.OutboxIT.(?) > >>>> org.kie.kogito.springboot.outbox.OutboxIT.(?) > >>>> org.kie.kogito.examples.OpenVinoRestIT.testOpenvino > >>>> > >>>> Regards > >>>> Jan > >>>> > >>>> On Thu, 5 Oct 2023 at 14:14, Jan Šťastný <[email protected]> > wrote: > >>>> > >>>> > Thanks to Roberto's efforts, the build-chain is fixed and released. > >>>> Many > >>>> > thanks. > >>>> > > >>>> > Taking Enrique's multi-repository PR as an example, build-chain > >>>> finally > >>>> > reports similar: > >>>> > --- > >>>> > # Checkout summary > >>>> > [INFO] [apache/incubator-kie-drools] > >>>> > [INFO] Project taken from apache/incubator-kie-drools:main (Branch > >>>> HEAD > >>>> > 1dda51ba5d4c60e293780fcbbdf736a1f52a4031) > >>>> > [INFO] Merged elguardian/drools:kie-issues-249 into branch main > >>>> > [INFO] [apache/incubator-kie-kogito-runtimes] > >>>> > [INFO] Project taken from apache/incubator-kie-kogito-runtimes:main > >>>> > (Branch HEAD 437a3167560b5260ec949f9e126967a91a00155d) > >>>> > [INFO] Merged elguardian/kogito-runtimes:kie-issues-249 into branch > >>>> main > >>>> > [INFO] [apache/incubator-kie-kogito-apps] > >>>> > [INFO] Project taken from apache/incubator-kie-kogito-apps:main > >>>> (Branch > >>>> > HEAD 73c27d909b577c2a21782deef91e68cbc2301b85) > >>>> > [INFO] Merged elguardian/kogito-apps:kie-issues-249 into branch main > >>>> > [INFO] [apache/incubator-kie-kogito-examples] > >>>> > [INFO] Project taken from apache/incubator-kie-kogito-examples:main > >>>> > (Branch HEAD 8663ef433056a278ae2657258f0ab366edc9abc5) > >>>> > [INFO] Merged elguardian/kogito-examples:kie-issues-249 into branch > >>>> main > >>>> > --- > >>>> > > >>>> > It does require retriggering the PR checks, which I have started to > >>>> do. > >>>> > > >>>> > There are numerous cases of old/stale PRs that need to be rebased or > >>>> > closed though. Please review your projects. > >>>> > > >>>> > Regards > >>>> > Jan > >>>> > > >>>> > On Thu, 5 Oct 2023 at 12:38, Francisco Javier Tirado Sarti < > >>>> > [email protected]> wrote: > >>>> > > >>>> >> I forgot to copy past the link of the PR > >>>> >> https://github.com/apache/incubator-kie-kogito-runtimes/pull/3227 > >>>> >> > >>>> >> On Thu, Oct 5, 2023 at 11:00 AM Francisco Javier Tirado Sarti < > >>>> >> [email protected]> wrote: > >>>> >> > >>>> >> > Hi Jan, > >>>> >> > I just rebased this one (one of the 5 I created pending on > >>>> runtimes), > >>>> >> but > >>>> >> > I cannot see the jenkins pipeline being executed (the gha failing > >>>> is > >>>> >> > expected) > >>>> >> > Am I missing something? > >>>> >> > > >>>> >> > On Thu, Oct 5, 2023 at 10:55 AM Jan Šťastný < > [email protected] > >>>> > > >>>> >> wrote: > >>>> >> > > >>>> >> >> Thanks for bringing that up. > >>>> >> >> > >>>> >> >> You're obviously right, I didn't account for the fact that the > >>>> >> build-chain > >>>> >> >> tool handles all on its own.We're blocked until the PR above is > >>>> merged > >>>> >> and > >>>> >> >> a new release in NPM appears. Given that it would be a bugfix I > >>>> suspect > >>>> >> >> existing version specifier would take the latest available: > >>>> >> >> > >>>> >> >> kiegroup/github-action-build-chain@v3 > >>>> >> >> > >>>> >> >> which is derived using > >>>> >> >> > >>>> >> >> > >>>> >> > >>>> > https://github.com/kiegroup/jenkins-pipeline-shared-libraries/blob/main/vars/buildChain.groovy > >>>> >> >> from > >>>> >> >> > >>>> >> >> > >>>> >> > >>>> > https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/actions/build-chain/action.yml#L36 > >>>> >> >> . > >>>> >> >> > >>>> >> >> Sorry for the mistake in summary and guidance. So merging is not > >>>> safe > >>>> >> >> judging on the PR checks results until we switch to updated > >>>> >> build-chain. > >>>> >> >> > >>>> >> >> Regards > >>>> >> >> Jan > >>>> >> >> > >>>> >> >> On Thu, 5 Oct 2023 at 10:45, Marek Novotny <[email protected] > > > >>>> >> wrote: > >>>> >> >> > >>>> >> >> > yes, that is correct Toshiya. For some reason the different > >>>> named > >>>> >> forked > >>>> >> >> > repository is not correctly check out by Build chain used in > CI > >>>> job. > >>>> >> >> > > >>>> >> >> > The PR > >>>> >> https://github.com/kiegroup/github-action-build-chain/pull/462 > >>>> >> >> > should fix that and then all CI jobs need to upgrade to new > >>>> version > >>>> >> of > >>>> >> >> BC > >>>> >> >> > to run with that fixed version of BC. > >>>> >> >> > > >>>> >> >> > BR, > >>>> >> >> > Dne 10/5/23 v 09:59 Toshiya Kobayashi napsal(a): > >>>> >> >> > > >>>> >> >> > Thank you very much for the progress, Jan and the people > >>>> involved! > >>>> >> >> > > >>>> >> >> > One question: > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > We've already seen green PR checks in incubator-kie-drools > >>>> >> >> > > >>>> >> >> > I raised a PR and checked the Jenkins CI yesterday. > >>>> >> >> > https://github.com/apache/incubator-kie-drools/pull/5545 > >>>> >> >> > > >>>> >> >> > ``` > >>>> >> >> > # Checkout summary > >>>> >> >> > [INFO] [apache/incubator-kie-drools] > >>>> >> >> > [INFO] Project taken from apache/incubator-kie-drools:main > >>>> (Branch > >>>> >> >> HEAD > >>>> >> >> > 02d592c720e43ca028d56e22375a750a9097f694) > >>>> >> >> > [INFO] [apache/incubator-kie-kogito-runtimes] > >>>> >> >> > [INFO] Project taken from > >>>> apache/incubator-kie-kogito-runtimes:main > >>>> >> >> > (Branch HEAD 2103ef90832274fb2aeeb55b6e5180263d2ea017) > >>>> >> >> > [INFO] [apache/incubator-kie-kogito-apps] > >>>> >> >> > [INFO] Project taken from > >>>> apache/incubator-kie-kogito-apps:main > >>>> >> >> (Branch > >>>> >> >> > HEAD 507ede90bd186f3f6e7fba8454faf19967acfc1f) > >>>> >> >> > [INFO] [apache/incubator-kie-kogito-examples] > >>>> >> >> > [INFO] Project taken from > >>>> apache/incubator-kie-kogito-examples:main > >>>> >> >> > (Branch HEAD 868814b52c2ea2caadb839f4652a4cc26fe7912a) > >>>> >> >> > ``` > >>>> >> >> > > >>>> >> >> > It didn't pick my PR branch > >>>> >> `tkobayas/drools:DROOLS-7570-test-hang-2`. > >>>> >> >> (my > >>>> >> >> > fork repo name is `drools`, not `incubator-kie-drools`) > >>>> >> >> > > >>>> >> >> > So don't we need to wait forhttps:// > >>>> >> >> github.com/kiegroup/github-action-build-chain/pull/462 even > for a > >>>> >> >> > single repo PR? > >>>> >> >> > > >>>> >> >> > Thanks! > >>>> >> >> > Toshiya > >>>> >> >> > > >>>> >> >> > On Thu, Oct 5, 2023 at 3:39 AM Jan Šťastný < > >>>> [email protected]> < > >>>> >> >> [email protected]> wrote: > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > Hello, > >>>> >> >> > we've managed to progress in some areas. Small steps. > >>>> >> >> > > >>>> >> >> > [PRs] > >>>> >> >> > We've already seen green PR checks in incubator-kie-drools and > >>>> >> >> > incubator-kie-kogito-runtimes repositories. If you have a > >>>> pending PR, > >>>> >> >> make > >>>> >> >> > sure you rebase with the latest changes from main, PR checks' > >>>> >> behavior > >>>> >> >> > depends on that. > >>>> >> >> > > >>>> >> >> > [Nightly] > >>>> >> >> > Recent problems with triggers resolved, tonight should trigger > >>>> first > >>>> >> >> drools > >>>> >> >> > nightly, to confirm the setup, other repos will be enabled > >>>> after. > >>>> >> >> > > >>>> >> >> > With this we've already found several issues, for some we > don't > >>>> have > >>>> >> >> > workaround yet. > >>>> >> >> > > >>>> >> >> > Most problematic is our inability to run multi-repository > >>>> builds for > >>>> >> >> PRs, > >>>> >> >> > so if we have changes related to each other in 3 related > >>>> repositories > >>>> >> >> they > >>>> >> >> > are not at the moment tested together. Which is a big problem > >>>> that is > >>>> >> >> being > >>>> >> >> > tackled already inhttps:// > >>>> >> >> github.com/kiegroup/github-action-build-chain/pull/462 . > >>>> >> >> > > >>>> >> >> > We're also seeing docker-related issues in execution of > >>>> >> >> > incubator-kie-kogito-apps and incubator-kie-kogito-examples PR > >>>> >> checks, > >>>> >> >> in > >>>> >> >> > most cases due to keycloak containers not being started - > anyone > >>>> >> >> willing to > >>>> >> >> > take a look, please jump on that, ping me if needed. > >>>> >> >> > > >>>> >> >> > We've also turned everything possible off, and we need to > >>>> gradually > >>>> >> put > >>>> >> >> > things back to shape. For immediate tasks I would name Apache > >>>> Nexus > >>>> >> >> > snapshot deployment, SonarCloud reporting and slightly > related - > >>>> >> >> > fine-grained PR checks. > >>>> >> >> > > >>>> >> >> > Regards > >>>> >> >> > Jan > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > -- > >>>> >> >> > Marek Novotny > >>>> >> >> > -- > >>>> >> >> > > >>>> >> >> > RedHat JBoss Middleware > >>>> >> >> > > >>>> >> >> > Red Hat Czech s.r.o. > >>>> >> >> > Purkynova 111 > >>>> >> >> > 612 45 Brno > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > >>>> >> > > >>>> >> > >>>> > > >>>> > >>> >
