Thank you, Tiago, for starting this thread and sharing the context with a detailed plan.
On Thu, Feb 22, 2024 at 8:29 PM Tiago Bento <[email protected]> wrote: > DISCLAIMER: this is a long email. Feel free to jump to the bottom for > the concrete task list. > > --- > > This email aims to explain the issue and lay out the efforts that are > in place to fix it. > > --- > > Before we start digging into the problem, we need to understand a > little bit of the history of the KIE community and the Kogito > initiative, at least from the part I started contributing to it anyway > :) > > Back in Business Central days, we had a single stream of development. > We had robust CI jobs that would build the entire stack, from `drools` > to `kie-wb-distributions`. When Kogito came about, there was a fission > in our pipelines. Kogito started to release independently from > everything else, and because of the organic way things started to > organize themselves, the Kogito Tooling initiative needed to have its > own release cycle as well. > > Up until we started the migration to Apache, Business Central > continued on its `7.x` stream, Kogito was at `1.x` stream, and Kogito > Tooling (which was already called KIE Tools) was still on `0.x`. > Drools started its own release stream as well, with `8.x`, and > OptaPlanner I believe had jumped to `9.x` already. This fragmentation > had technical implications, which we are trying to overcome right now > with Apache KIE 10. > > The Apache KIE 10 release is aimed to be an atomic release, where the > entire Apache KIE community follows a single release stream. This > means we need to make sure our code base is in sync, from `drools` to > `kie-tools`. > > This whole conversation of course doesn't come without a background of > CI pipelines, organizational perspectives of what Apache KIE is, and > even poly- vs. mono-repo discussions. This is not the focus of this > issue, and I believe with time we'll converge into something everyone > is comfortable with and that ultimately serves our users well. > > --- > > Due to the lack of unification and flexibility while bootstrapping new > efforts, KIE Tools had to come up with a way to depend on the new > stuff (i.e. Kogito) without sacrificing its own integrity and > stability. KIE Tools, as many of you know, evolved to be a polyglot > monorepo, hosting Java, JavaScript/TypeScript, Go and Docker images. > There's a multitude of technologies at play on KIE Tools, which > ultimately gives our users KIE Sandbox, VS Code and Chrome Extensions, > and Standalone Editors that can be used as regular libraries on web > applications. > > The technical solution we found on `kie-tools` was depending on Kogito > as if it were a 3rd party dependency. We have a single, fixed version > of it and we manually upgrade it to the latest one available when we > need to. This has worked amazingly well, allowing us to own the > codebase and avoid random breaks coming from SNAPSHOT artifacts. > > This allowed the `drools` and `kogito-*` repositories to evolve > independently of `kie-tools`. This also made `kie-tools` responsible > for adapting itself to changes coming from `drools` and `kogito-*`. > Luckily, we didn't have many incompatibilities, but right now we have > an important one, that we'll discuss below. > > Note that at this point, and for this issue in particular, it is > unimportant to discuss why things were done the way they were, or what > we should've done instead. I'm laying out the facts so that we can > understand how we're moving forward now. Of course, I would personally > love to debate this on a separate thread :) > > --- > > One of the most impactful features of KIE Sandbox is the DMN Runner. > It allows people to rapidly interact with their Decisions as they > author them, and this was all made possible because of the JIT > Executor (hosted at `kogito-apps`). KIE Sandbox also started to allow > users to deploy their Decisions to OpenShift, then later to > Kubernetes, and now recently we allow users to deploy anything to > OpenShift and Kubernetes, with a very high level of customization. All > these features depend on `kogito-*`, Quarkus, and all the things that > come with them. Including the Java version. > > Logically, `kie-tools` depends on `drools` and `kogito-*`, as `drools` > and `kogito-*` can (and do!) exist and function without Tools, but not > the other way around. > > As `kie-tools` started growing, maturing, and releasing some pieces of > technology that were useful to other domains (i.e. Multiplying > Architecture), it became a dependency for other projects. Users > started depending on our Standalone Editors, but also on the > Multiplying Architecture packages. One good example is Kaoto, which > released a VS Code extension aligned with our architecture. > > The capability of encapsulating web components in any technology > behind a well defined, statically typed API was really appealing, and > when the Kogito initiative started to design their Runtime Tools > (i.e., Quarkus Dev UIs, Management Console, and Task Inbox), using the > Multiplying Architecture was the way to go. > > That's when the circular dependency started. We had > `kogito-apps/ui-packages` depending on `kie-tools`. Note that it was > also logical for these packages to be part of the `kogito-*` > repositories and release stream. Users upgrading to a new version of > Kogito would also need to have their Task Inbox, Management Console, > and Quarkus Dev UIs aligned with the new version of Kogito. Not to > mention Job Service, Data Index etc. > > For many months (or years!), this was not an issue, as the Multiplying > Architecture packages were very stable and very rarely were modified. > The last big change we had was 3 years ago, with the introduction of > Shared Values [1], which did not impact the fire-and-forget and > Promise-based mechanisms we already had in place. > > Although we were aware of this circular dependency from the beginning, > it was not that big of a deal for `kogito-apps` to depend on an older, > dephased version of `kie-tools`. We didn't have plans of unifying the > streams anyway, and it was the best solution for our users to have > their Runtime Tools packages released together with the Kogito stream. > > That of course changed after the migration to Apache. > > --- > > If you reached this point, thank you for your interest in a little bit > of the history of KIE, and I hope you can now understand why things > are the way they are. Let's talk about what changed and what we are > doing to adapt to the new requirements. > > Hinting a little bit to what I mentioned above, this discussion is > very intertwined with the CI pipelines and versioning/releasing > strategies subjects, as we are essentially discussing code > organization, coupling/decoupling and granularity of stuff, and to > make things easier at least for a while, `drools` and `kogito-*` will > continue being a separate stream from `kie-tools`. > > `drools` and `kogito-*` will continue being able to evolve > independently of `kie-tools`, and `kie-tools` will continue adapting > itself to whatever changes on `drools` and `kogito-*`. > > At the same time, Apache KIE 10 (and the next releases too!) will be > atomic. And for our users, a single release stream. `drools@10` and > `kogito-*@10` will be used on `kie-tools@10`. We started this effort a > while ago, and although we should've been more open about it from the > beginning, we are also learning how to properly communicate inside the > Apache structure. Better late than never :) Thanks for the tip Tibor! > > Here's how we'll do it: > > 1. A couple of weeks ago, one of our newest committers. Rodrigo > Antunes, created automated jobs to weekly publish a timestamped > SNAPSHOT version of `drools` and `kogito-*`. This will allow us to > have a persistent, immutable version to point to on `kie-tools`, even > without having a full release. Timestamped SNAPSHOTs, as we're calling > them, are a little bit different than normal Maven SNAPSHOTs, as there > is no `updatePolicy`, so as time passes, the downloaded artifacts do > not change. This allows `kie-tools` to be stable and incorporate > changes done to `drools` and `kogito-*` at its own pace. We also moved > the publication of `@kie-tools/jitexecutor-native` to this new job, so > we won't have to maintain it on the `kie-tools` side anymore. Thanks > Rodrigo! > > 2. When we define the code freeze prior to releasing Apache KIE, > `drools` and `kogito-*` will need to be frozen a little bit earlier > than `kie-tools`, leaving a buffer for `kie-tools` to start pointing > to the latest timestamped SNAPSHOT and adapting itself to the changes. > My initial thoughts are around 2 weeks, but with constant updates and > proper alignment between the two streams, I see this time reducing to > a couple of days for upcoming releases. > > 3. After `kie-tools` is fully functional pointing to the latest > timestamped SNAPSHOT of `drools` and `kogito-*`, we can now start > preparing for the actual release. We will cut new branches for the > next version (i.e. 10.0.0) and update all configuration files that we > have to. This will be done on `drools`, `kogito-*`, and `kie-tools`. > Every module/package will declare the same version, and will depend on > the upstream modules/packages with the same version as well. We can > proceed with this staging phase as normal. Running automated/manual > tests and eventually signing off that everything is good to go. (I'm > not 100% on the timeline for the Apache release, but with this process > we can guarantee that we will vote on branches that won't change when > we decide to release. It will be a matter of pushing tags pointing to > these release branches). > > With the release process defined, we can now discuss the exact actions > we're taking to remove the circular dependency we have between > `kie-tools` and `kogito-apps`. All of which will need to be addressed > prior to starting the release process. > > 1. (Fabrizio - IN PROGRESS) Make sure `kie-tools` is on Java 17, Maven > 3.9.6, and Quarkus 3. > (https://github.com/apache/incubator-kie-issues/issues/960) > There's a complication here, since we still have three big > packages (Serverless Workflow Diagram Editor, BPMN, DMN, and SceSim > Editors, and DashBuilder) on `kie-tools` that need GWT, and as you > know, GWT together with Java upgrades usually require some work. We're > on GWT 2.10, which should already support Java 17, but we'll know more > once we try it. > > 2. (Fabrizio - NOT YET STARTED) Make sure `kie-tools` is using the > latest timestamped SNAPSHOT from `kogito-*`. > (https://github.com/apache/incubator-kie-issues/issues/965) > We have them built and published every Sunday EOD, so on Monday > (Feb 25th, 2024) we'll have a brand new one to start working with. > > 3. (Thiago - IN PROGRESS) Move `kogito-apps/ui-packages` to > `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/833) > We started a couple of weeks ago, and we should be ~80% done with it > by now. > > 4. (Fabrizio - PR SENT, but blocked by 1. and 2.) Move the SWF Quarkus > Dev UI package from `kogito-apps` to `kie-tools`. > (https://github.com/apache/incubator-kie-tools/pull/2167) > This can only be done after 1. and 2. are merged, since this move > will require `kie-tools` to be ready to build packages with Java 17, > Maven 3.9.6, and Quarkus 3. > > 5. (Thiago - NOT YET STARTED) Move the BPMN Quarkus Dev UI package > from `kogito-apps` to `kie-tools`. > (https://github.com/apache/incubator-kie-issues/issues/966) > We'll follow the lead of Fabrizio's PR from 4. That's essentially > doing exactly the same thing for the BPMN Quarkus Dev UI module. > > 6. (Rodrigo - NOT YET STARTED) Change the Jenkins release jobs to > include publishing both Quarkus Dev UI modules to Maven central as > part of the Apache release. > (https://github.com/apache/incubator-kie-issues/issues/964) > Should be pretty straight-forward, as we already have experience > doing this for other repos. > > 7. (Thiago - NOT YET STARTED) Change the ownership of `task-console` > and `management-console` images from `kogito-images` to `kie-tools`. > (https://github.com/apache/incubator-kie-issues/issues/963) > KIE Tools already hosts and publishes several images, and since > the content of those images will now be hosted at `kie-tools`, we can > only host them there. > This also means we'll get rid of the two Quarkus apps [2] that > wrap both consoles' static assets. We could be wrong about the purpose > of those apps, so if anyone has more information, please do reach out. > Otherwise, we'll skip the Quarkusification of those static assets and > we'll ship images with `httpd` directly, like we do for KIE Sandbox. > > 8. (Assignee TDB - NOT YET STARTED) Deleting everything that was moved > to `kie-tools` from `kogito-apps`. > (https://github.com/apache/incubator-kie-issues/issues/962) > That will be the point where we actually get rid of the circular > dependency and remove the blocker for the release. > > 9. (Assignee TDB - NOT YET STARTED) Adapt `kogito-examples` to all the > changes we mentioned above. > (https://github.com/apache/incubator-kie-issues/issues/961) > Kogito Examples keeps being part of the `drools` and `kogito-*` > stream, meaning that example modules won't be able to use neither of > the Quarkus Dev UIs, as `kie-tools` is downstream in relation to the > `drools` and `kogito-*` stream. > The same would be true for Management Console and Task Console > images, but for them, we can use the `daily-dev` tag we publish daily > from `kie-tools@main`. > > 10. (Assignee TDB - NOT YET STARTED) Create some example apps that use > the Quarkus Dev UIs directly on the `examples` directory of > `kie-tools`. (https://github.com/apache/incubator-kie-issues/issues/959) > This is to serve the same purpose of packages that did that on > `kogito-examples`, but now hosted on `kie-tools`. > > Pheew! That's a lot of things to do, but it is the result of two > intense weeks assessing and planning how we would address this issue > in a way that the Apache KIE 10 release can be done in a way that > feels atomic to our users, who are ultimately the most important part > of the community. There's an open EPIC too at > https://github.com/apache/incubator-kie-issues/issues/967. > > I appreciate it if you reached this point, and hope you feel welcome > to contributing to solving this lengthy and historical problem if you > feel like doing so. > > I'll personally send updates to this thread every week, on Thursday > nights. Do not hesitate to contact me or any of the people involved in > this effort if you have questions, or simply want to chat about it. > > Thanks a lot Alex, Paulo, Fabrizio, Thiago, Rodrigo, and others that > somehow helped so far. > > Kind regards, > > Tiago Bento > > --- > > P.S.: You might've noticed we left Trusty out of the picture for now. > Thiago is conducting an investigation to understand what would be the > best course of action for it. As much as we have plans to revive it, > right now our focus has to be on getting Apache KIE 10 out. We can > always rely on Git to bring it back in the future, if we ultimately > decide that wiping out Trusty is the best strategy for this moment. > > [1] https://github.com/apache/incubator-kie-tools/pull/691 > > [2] > https://github.com/apache/incubator-kie-kogito-apps/tree/main/task-console > and > https://github.com/apache/incubator-kie-kogito-apps/tree/main/management-console > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
