Dominik, The plans and course of action for removing the circular dependency has been exhaustively discussed in the mailing list and Zulip channel among community and key stakeholders, I don’t think we have room to revisit the decision that has been already agreed.
With that being said, I don’t think the current plans removes any functionality, we are just moving things from one repository to another. On Wed, Mar 6, 2024 at 2:19 AM Dominik Hanak <[email protected]> wrote: > Hi Tiago, > > that is a problem, we are removing an important functionality for our SWF > users, imho it is a killer feature in this case. > I am against it, there needs to be a solution where we can depend on > kie-tools with kie-kogito-images. > > Another thing that is currently open as a PR [2] that makes images depend > on kie-tools. Here we are following > the approach you started in kiegroup and we were not aware of any changes, > now with the unexpected and sudden changes it is irrelevant? > Do you have a proposal on how to achieve what we have in [2] after the > circular dependency issue is fixed according to the proposal annoucement? > > Dominik > > > [2] https://github.com/apache/incubator-kie-kogito-images/pull/1734 > > > On Wed, 6 Mar 2024 at 02:10, Tiago Bento <[email protected]> > wrote: > > > Hi Dominik, > > > > That is indeed an oversight on our part. Thanks for pointing that out. > > > > Unfortunately, given the two development streams (`drools` and > > `kogito-*`, and `kie-tools`), we can't have `kogito-images` depending > > on packages that will start being hosted on `kie-tools`, since > > `kogito-images` is part of the `drools` and `kogito-*` development > > stream, which is upstream from `kie-tools`' perspective. > > > > Since all Dev UI packages are moving to `kie-tools`, images on > > `kogito-images` can't depend on them. > > > > I'm not familiar with the structure of `kogito-images`, nor the > > purpose of this image that uses SWF's Quarkus Dev UI, but following > > the same reasoning we did for `kogito-management-console` and > > `kogito-task-console`, my suggestion would be moving this image to > > `kie-tools`. Another option I can see is simply removing the Quarkus > > Dev UI dependency on this image. > > > > I'd wait for other opinions, though, as I'm not the best person to > > answer SWF-related questions. > > > > Regards, > > > > Tiago Bento > > > > > > On Tue, Mar 5, 2024 at 9:31 AM Dominik Hanak <[email protected]> > > wrote: > > > > > > Hi Tiago, > > > > > > Thanks for moving this forward. > > > > > > I have one thing to highlight with > > > https://github.com/apache/incubator-kie-kogito-images > > > I noticed [1], where the swf devui is being used in an image. Imho that > > > would require adjustments to the nightly pipelines & release process. > > > Would it be possible to add tasks to the plan that ensure required > > > adjustments to our CI once the migration is done? > > > > > > Best regards, > > > > > > Dominik > > > > > > [1] > > > > > > https://github.com/apache/incubator-kie-kogito-images/blob/ce62b9d86c2ea0385ab85c3ef5339afe5ecdea72/modules/kogito-swf/devmode/build-config/module.yaml#L30 > > > > > > > > > On Fri, 1 Mar 2024 at 05:54, Tiago Bento <[email protected]> > wrote: > > > > > > > Hi everyone, > > > > > > > > Following up on the status of this effort. > > > > > > > > I know I'm a bit late on the email, but it paid off, as we just had > > > > our first green build [1] of KIE Tools with Java 17, Quarkus 3.2.9, > > > > Maven 3.9.6 and Kogito 999-20240218-SNAPSHOT. We ended up deciding to > > > > do items 1. and 2. together after all, but we couldn't use the latest > > > > timestamped SNAPSHOT due to > > > > https://issues.apache.org/jira/browse/INFRA-25546, which prevented > the > > > > publication of the timestamped SNAPSHOT to succeed. That's why we > > > > picked 20240218, not 20240225. > > > > > > > > We will proceed with some manual checks now to confirm that > everything > > > > is working properly, and I hope on the next status update we will > have > > > > the big blocker removed and that we have already made some progress > on > > > > items other than 1. and 2.. > > > > > > > > It's been great seeing everyone working together to move this forward > > > > as fast as possible. Thanks a lot Eder, Thiago, Rodrigo, Alex, > > > > Fabrizio, Paulo, and Yeser! > > > > > > > > [1] > > > > > > > https://github.com/apache/incubator-kie-tools/actions/runs/8105498131/job/22153897965 > > > > > > > > On Mon, Feb 26, 2024 at 5:59 AM Francisco Javier Tirado Sarti > > > > <[email protected]> wrote: > > > > > > > > > > Hi Tiago, > > > > > It was a nice morning reading ;) Thanks for all the time invested > on > > such > > > > > detailed writing. > > > > > @Ricardo Zanini <[email protected]> @Walter Medvedeo > > > > > <[email protected]> I guess dev ui can be removed from > > current > > > > > examples, at least from the ones I'm familiar with, since their > > README > > > > > files do not really require any graphical tool. But to be honest, > > since I > > > > > have focused my work on SWF simple showcase examples, I'm not 100% > > sure > > > > > that's the same case for more elaborate ones, can you please check > > that > > > > it > > > > > is fine to remove dev ui? If this is not the case, maybe those > > examples > > > > can > > > > > be moved to the tools repository (points 9 and 10 of Tiago's email) > > > > > Thanks in advance. > > > > > > > > > > On Fri, Feb 23, 2024 at 2:29 AM Tiago Bento <[email protected] > > > > > > wrote: > > > > > > > > > > > DISCLAIMER: this is a long email. Feel free to jump to the bottom > > for > > > > > > the concrete task list. > > > > > > > > > > > > --- > > > > > > > > > > > > This email aims to explain the issue and lay out the efforts that > > are > > > > > > in place to fix it. > > > > > > > > > > > > --- > > > > > > > > > > > > Before we start digging into the problem, we need to understand a > > > > > > little bit of the history of the KIE community and the Kogito > > > > > > initiative, at least from the part I started contributing to it > > anyway > > > > > > :) > > > > > > > > > > > > Back in Business Central days, we had a single stream of > > development. > > > > > > We had robust CI jobs that would build the entire stack, from > > `drools` > > > > > > to `kie-wb-distributions`. When Kogito came about, there was a > > fission > > > > > > in our pipelines. Kogito started to release independently from > > > > > > everything else, and because of the organic way things started to > > > > > > organize themselves, the Kogito Tooling initiative needed to have > > its > > > > > > own release cycle as well. > > > > > > > > > > > > Up until we started the migration to Apache, Business Central > > > > > > continued on its `7.x` stream, Kogito was at `1.x` stream, and > > Kogito > > > > > > Tooling (which was already called KIE Tools) was still on `0.x`. > > > > > > Drools started its own release stream as well, with `8.x`, and > > > > > > OptaPlanner I believe had jumped to `9.x` already. This > > fragmentation > > > > > > had technical implications, which we are trying to overcome right > > now > > > > > > with Apache KIE 10. > > > > > > > > > > > > The Apache KIE 10 release is aimed to be an atomic release, where > > the > > > > > > entire Apache KIE community follows a single release stream. This > > > > > > means we need to make sure our code base is in sync, from > `drools` > > to > > > > > > `kie-tools`. > > > > > > > > > > > > This whole conversation of course doesn't come without a > > background of > > > > > > CI pipelines, organizational perspectives of what Apache KIE is, > > and > > > > > > even poly- vs. mono-repo discussions. This is not the focus of > this > > > > > > issue, and I believe with time we'll converge into something > > everyone > > > > > > is comfortable with and that ultimately serves our users well. > > > > > > > > > > > > --- > > > > > > > > > > > > Due to the lack of unification and flexibility while > bootstrapping > > new > > > > > > efforts, KIE Tools had to come up with a way to depend on the new > > > > > > stuff (i.e. Kogito) without sacrificing its own integrity and > > > > > > stability. KIE Tools, as many of you know, evolved to be a > polyglot > > > > > > monorepo, hosting Java, JavaScript/TypeScript, Go and Docker > > images. > > > > > > There's a multitude of technologies at play on KIE Tools, which > > > > > > ultimately gives our users KIE Sandbox, VS Code and Chrome > > Extensions, > > > > > > and Standalone Editors that can be used as regular libraries on > web > > > > > > applications. > > > > > > > > > > > > The technical solution we found on `kie-tools` was depending on > > Kogito > > > > > > as if it were a 3rd party dependency. We have a single, fixed > > version > > > > > > of it and we manually upgrade it to the latest one available when > > we > > > > > > need to. This has worked amazingly well, allowing us to own the > > > > > > codebase and avoid random breaks coming from SNAPSHOT artifacts. > > > > > > > > > > > > This allowed the `drools` and `kogito-*` repositories to evolve > > > > > > independently of `kie-tools`. This also made `kie-tools` > > responsible > > > > > > for adapting itself to changes coming from `drools` and > `kogito-*`. > > > > > > Luckily, we didn't have many incompatibilities, but right now we > > have > > > > > > an important one, that we'll discuss below. > > > > > > > > > > > > Note that at this point, and for this issue in particular, it is > > > > > > unimportant to discuss why things were done the way they were, or > > what > > > > > > we should've done instead. I'm laying out the facts so that we > can > > > > > > understand how we're moving forward now. Of course, I would > > personally > > > > > > love to debate this on a separate thread :) > > > > > > > > > > > > --- > > > > > > > > > > > > One of the most impactful features of KIE Sandbox is the DMN > > Runner. > > > > > > It allows people to rapidly interact with their Decisions as they > > > > > > author them, and this was all made possible because of the JIT > > > > > > Executor (hosted at `kogito-apps`). KIE Sandbox also started to > > allow > > > > > > users to deploy their Decisions to OpenShift, then later to > > > > > > Kubernetes, and now recently we allow users to deploy anything to > > > > > > OpenShift and Kubernetes, with a very high level of > customization. > > All > > > > > > these features depend on `kogito-*`, Quarkus, and all the things > > that > > > > > > come with them. Including the Java version. > > > > > > > > > > > > Logically, `kie-tools` depends on `drools` and `kogito-*`, as > > `drools` > > > > > > and `kogito-*` can (and do!) exist and function without Tools, > but > > not > > > > > > the other way around. > > > > > > > > > > > > As `kie-tools` started growing, maturing, and releasing some > > pieces of > > > > > > technology that were useful to other domains (i.e. Multiplying > > > > > > Architecture), it became a dependency for other projects. Users > > > > > > started depending on our Standalone Editors, but also on the > > > > > > Multiplying Architecture packages. One good example is Kaoto, > which > > > > > > released a VS Code extension aligned with our architecture. > > > > > > > > > > > > The capability of encapsulating web components in any technology > > > > > > behind a well defined, statically typed API was really appealing, > > and > > > > > > when the Kogito initiative started to design their Runtime Tools > > > > > > (i.e., Quarkus Dev UIs, Management Console, and Task Inbox), > using > > the > > > > > > Multiplying Architecture was the way to go. > > > > > > > > > > > > That's when the circular dependency started. We had > > > > > > `kogito-apps/ui-packages` depending on `kie-tools`. Note that it > > was > > > > > > also logical for these packages to be part of the `kogito-*` > > > > > > repositories and release stream. Users upgrading to a new version > > of > > > > > > Kogito would also need to have their Task Inbox, Management > > Console, > > > > > > and Quarkus Dev UIs aligned with the new version of Kogito. Not > to > > > > > > mention Job Service, Data Index etc. > > > > > > > > > > > > For many months (or years!), this was not an issue, as the > > Multiplying > > > > > > Architecture packages were very stable and very rarely were > > modified. > > > > > > The last big change we had was 3 years ago, with the introduction > > of > > > > > > Shared Values [1], which did not impact the fire-and-forget and > > > > > > Promise-based mechanisms we already had in place. > > > > > > > > > > > > Although we were aware of this circular dependency from the > > beginning, > > > > > > it was not that big of a deal for `kogito-apps` to depend on an > > older, > > > > > > dephased version of `kie-tools`. We didn't have plans of unifying > > the > > > > > > streams anyway, and it was the best solution for our users to > have > > > > > > their Runtime Tools packages released together with the Kogito > > stream. > > > > > > > > > > > > That of course changed after the migration to Apache. > > > > > > > > > > > > --- > > > > > > > > > > > > If you reached this point, thank you for your interest in a > little > > bit > > > > > > of the history of KIE, and I hope you can now understand why > things > > > > > > are the way they are. Let's talk about what changed and what we > are > > > > > > doing to adapt to the new requirements. > > > > > > > > > > > > Hinting a little bit to what I mentioned above, this discussion > is > > > > > > very intertwined with the CI pipelines and versioning/releasing > > > > > > strategies subjects, as we are essentially discussing code > > > > > > organization, coupling/decoupling and granularity of stuff, and > to > > > > > > make things easier at least for a while, `drools` and `kogito-*` > > will > > > > > > continue being a separate stream from `kie-tools`. > > > > > > > > > > > > `drools` and `kogito-*` will continue being able to evolve > > > > > > independently of `kie-tools`, and `kie-tools` will continue > > adapting > > > > > > itself to whatever changes on `drools` and `kogito-*`. > > > > > > > > > > > > At the same time, Apache KIE 10 (and the next releases too!) will > > be > > > > > > atomic. And for our users, a single release stream. `drools@10` > > and > > > > > > `kogito-*@10` will be used on `kie-tools@10`. We started this > > effort a > > > > > > while ago, and although we should've been more open about it from > > the > > > > > > beginning, we are also learning how to properly communicate > inside > > the > > > > > > Apache structure. Better late than never :) Thanks for the tip > > Tibor! > > > > > > > > > > > > Here's how we'll do it: > > > > > > > > > > > > 1. A couple of weeks ago, one of our newest committers. Rodrigo > > > > > > Antunes, created automated jobs to weekly publish a timestamped > > > > > > SNAPSHOT version of `drools` and `kogito-*`. This will allow us > to > > > > > > have a persistent, immutable version to point to on `kie-tools`, > > even > > > > > > without having a full release. Timestamped SNAPSHOTs, as we're > > calling > > > > > > them, are a little bit different than normal Maven SNAPSHOTs, as > > there > > > > > > is no `updatePolicy`, so as time passes, the downloaded artifacts > > do > > > > > > not change. This allows `kie-tools` to be stable and incorporate > > > > > > changes done to `drools` and `kogito-*` at its own pace. We also > > moved > > > > > > the publication of `@kie-tools/jitexecutor-native` to this new > > job, so > > > > > > we won't have to maintain it on the `kie-tools` side anymore. > > Thanks > > > > > > Rodrigo! > > > > > > > > > > > > 2. When we define the code freeze prior to releasing Apache KIE, > > > > > > `drools` and `kogito-*` will need to be frozen a little bit > earlier > > > > > > than `kie-tools`, leaving a buffer for `kie-tools` to start > > pointing > > > > > > to the latest timestamped SNAPSHOT and adapting itself to the > > changes. > > > > > > My initial thoughts are around 2 weeks, but with constant updates > > and > > > > > > proper alignment between the two streams, I see this time > reducing > > to > > > > > > a couple of days for upcoming releases. > > > > > > > > > > > > 3. After `kie-tools` is fully functional pointing to the latest > > > > > > timestamped SNAPSHOT of `drools` and `kogito-*`, we can now start > > > > > > preparing for the actual release. We will cut new branches for > the > > > > > > next version (i.e. 10.0.0) and update all configuration files > that > > we > > > > > > have to. This will be done on `drools`, `kogito-*`, and > > `kie-tools`. > > > > > > Every module/package will declare the same version, and will > > depend on > > > > > > the upstream modules/packages with the same version as well. We > can > > > > > > proceed with this staging phase as normal. Running > automated/manual > > > > > > tests and eventually signing off that everything is good to go. > > (I'm > > > > > > not 100% on the timeline for the Apache release, but with this > > process > > > > > > we can guarantee that we will vote on branches that won't change > > when > > > > > > we decide to release. It will be a matter of pushing tags > pointing > > to > > > > > > these release branches). > > > > > > > > > > > > With the release process defined, we can now discuss the exact > > actions > > > > > > we're taking to remove the circular dependency we have between > > > > > > `kie-tools` and `kogito-apps`. All of which will need to be > > addressed > > > > > > prior to starting the release process. > > > > > > > > > > > > 1. (Fabrizio - IN PROGRESS) Make sure `kie-tools` is on Java 17, > > Maven > > > > > > 3.9.6, and Quarkus 3. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/960) > > > > > > There's a complication here, since we still have three big > > > > > > packages (Serverless Workflow Diagram Editor, BPMN, DMN, and > SceSim > > > > > > Editors, and DashBuilder) on `kie-tools` that need GWT, and as > you > > > > > > know, GWT together with Java upgrades usually require some work. > > We're > > > > > > on GWT 2.10, which should already support Java 17, but we'll know > > more > > > > > > once we try it. > > > > > > > > > > > > 2. (Fabrizio - NOT YET STARTED) Make sure `kie-tools` is using > the > > > > > > latest timestamped SNAPSHOT from `kogito-*`. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/965) > > > > > > We have them built and published every Sunday EOD, so on > Monday > > > > > > (Feb 25th, 2024) we'll have a brand new one to start working > with. > > > > > > > > > > > > 3. (Thiago - IN PROGRESS) Move `kogito-apps/ui-packages` to > > > > > > `kie-tools`. ( > > > > https://github.com/apache/incubator-kie-issues/issues/833) > > > > > > We started a couple of weeks ago, and we should be ~80% done > > with > > > > it > > > > > > by now. > > > > > > > > > > > > 4. (Fabrizio - PR SENT, but blocked by 1. and 2.) Move the SWF > > Quarkus > > > > > > Dev UI package from `kogito-apps` to `kie-tools`. > > > > > > (https://github.com/apache/incubator-kie-tools/pull/2167) > > > > > > This can only be done after 1. and 2. are merged, since this > > move > > > > > > will require `kie-tools` to be ready to build packages with Java > > 17, > > > > > > Maven 3.9.6, and Quarkus 3. > > > > > > > > > > > > 5. (Thiago - NOT YET STARTED) Move the BPMN Quarkus Dev UI > package > > > > > > from `kogito-apps` to `kie-tools`. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/966) > > > > > > We'll follow the lead of Fabrizio's PR from 4. That's > > essentially > > > > > > doing exactly the same thing for the BPMN Quarkus Dev UI module. > > > > > > > > > > > > 6. (Rodrigo - NOT YET STARTED) Change the Jenkins release jobs to > > > > > > include publishing both Quarkus Dev UI modules to Maven central > as > > > > > > part of the Apache release. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/964) > > > > > > Should be pretty straight-forward, as we already have > > experience > > > > > > doing this for other repos. > > > > > > > > > > > > 7. (Thiago - NOT YET STARTED) Change the ownership of > > `task-console` > > > > > > and `management-console` images from `kogito-images` to > > `kie-tools`. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/963) > > > > > > KIE Tools already hosts and publishes several images, and > since > > > > > > the content of those images will now be hosted at `kie-tools`, we > > can > > > > > > only host them there. > > > > > > This also means we'll get rid of the two Quarkus apps [2] > that > > > > > > wrap both consoles' static assets. We could be wrong about the > > purpose > > > > > > of those apps, so if anyone has more information, please do reach > > out. > > > > > > Otherwise, we'll skip the Quarkusification of those static assets > > and > > > > > > we'll ship images with `httpd` directly, like we do for KIE > > Sandbox. > > > > > > > > > > > > 8. (Assignee TDB - NOT YET STARTED) Deleting everything that was > > moved > > > > > > to `kie-tools` from `kogito-apps`. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/962) > > > > > > That will be the point where we actually get rid of the > > circular > > > > > > dependency and remove the blocker for the release. > > > > > > > > > > > > 9. (Assignee TDB - NOT YET STARTED) Adapt `kogito-examples` to > all > > the > > > > > > changes we mentioned above. > > > > > > (https://github.com/apache/incubator-kie-issues/issues/961) > > > > > > Kogito Examples keeps being part of the `drools` and > `kogito-*` > > > > > > stream, meaning that example modules won't be able to use neither > > of > > > > > > the Quarkus Dev UIs, as `kie-tools` is downstream in relation to > > the > > > > > > `drools` and `kogito-*` stream. > > > > > > The same would be true for Management Console and Task > Console > > > > > > images, but for them, we can use the `daily-dev` tag we publish > > daily > > > > > > from `kie-tools@main`. > > > > > > > > > > > > 10. (Assignee TDB - NOT YET STARTED) Create some example apps > that > > use > > > > > > the Quarkus Dev UIs directly on the `examples` directory of > > > > > > `kie-tools`. ( > > > > https://github.com/apache/incubator-kie-issues/issues/959) > > > > > > This is to serve the same purpose of packages that did that > on > > > > > > `kogito-examples`, but now hosted on `kie-tools`. > > > > > > > > > > > > Pheew! That's a lot of things to do, but it is the result of two > > > > > > intense weeks assessing and planning how we would address this > > issue > > > > > > in a way that the Apache KIE 10 release can be done in a way that > > > > > > feels atomic to our users, who are ultimately the most important > > part > > > > > > of the community. There's an open EPIC too at > > > > > > https://github.com/apache/incubator-kie-issues/issues/967. > > > > > > > > > > > > I appreciate it if you reached this point, and hope you feel > > welcome > > > > > > to contributing to solving this lengthy and historical problem if > > you > > > > > > feel like doing so. > > > > > > > > > > > > I'll personally send updates to this thread every week, on > Thursday > > > > > > nights. Do not hesitate to contact me or any of the people > > involved in > > > > > > this effort if you have questions, or simply want to chat about > it. > > > > > > > > > > > > Thanks a lot Alex, Paulo, Fabrizio, Thiago, Rodrigo, and others > > that > > > > > > somehow helped so far. > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > Tiago Bento > > > > > > > > > > > > --- > > > > > > > > > > > > P.S.: You might've noticed we left Trusty out of the picture for > > now. > > > > > > Thiago is conducting an investigation to understand what would be > > the > > > > > > best course of action for it. As much as we have plans to revive > > it, > > > > > > right now our focus has to be on getting Apache KIE 10 out. We > can > > > > > > always rely on Git to bring it back in the future, if we > ultimately > > > > > > decide that wiping out Trusty is the best strategy for this > moment. > > > > > > > > > > > > [1] https://github.com/apache/incubator-kie-tools/pull/691 > > > > > > > > > > > > [2] > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-apps/tree/main/task-console > > > > > > and > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-apps/tree/main/management-console > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
