Back when we started this thread, the concern was that moving the SonataFlow Operator to `kie-tools` would be permanent. We explicitly agreed that this would not be the case, and we created https://github.com/apache/incubator-kie-issues/issues/1040 to solidify that consensus. More than that, we have the pull request sent, fulfilling the agreement.
The current state of the Apache KIE codebase is somewhat reasonable, IMHO, although not approved by everyone, with the SonataFlow Images and Operator inside `kie-tools`. Worth noting, however, that currently, as Walter mentioned, we have a working setup (for both 10.0.x and main streams) with all the repos that are part of our release strategy having automations around CI, daily/weekly publishing jobs, and aligned with the release automations. The place where we started (i.e., before moving the SonataFlow Images and Operator into `kie-tools`) was a dysfunctional codebase, with cyclic dependencies, references to old versions, dead artifacts, you name it. With a lot of dedication and hard work from many people, we managed to overcome all that and branch 10.0.x, moving closer and closer to the 10.0.0 release. Past 10.0.0, though, we would revert that move (as agreed), and go back to a semi-dysfunctional codebase. Don't get me wrong, we're much better now than we ever were, IMHO, but as others are saying, we don't have a reasonable plan for the future yet. With all that said, there's an important parenthesis here. That is the fact that the `incubator-kie-serverless-operator` continued existing AND evolving, contrary to what I believed would be the case, as the source of truth for Apache KIE 10 for the SonataFlow Images and Operator is `kie-tools`. I don't know what landed on `kie-tools`, and what landed on `incubator-kie-serverless-operator`, but I know that having both is not the way to go. IMHO, now that the 10.0.x stream exists, and `incubator-kie-serverless-operator` is not a part of it, we need to make a call if we want to stick to the agreement that we had, or if we want to do something else. @Walter All your points make sense to me, and we tried to explain all this during the conversation before we had the strategy for 10.0.0 defined. I'm glad you find that we have a working setup, but I don't think we can continue with this duplication for the SonataFlow Operator codebase. On Thu, Jul 18, 2024 at 10:12 AM Walter Medvedeo <[email protected]> wrote: > > Hi Tibor, > > What we are holding for two more weeks is not the release. > What we need is some more time for, is to kill the build process of 3 images, > that we produce from main, with a procedure that already exits and is working > with no issues. > > On 2024/07/18 13:40:39 Tibor Zimányi wrote: > > Hi, > > > > before holding for another two weeks, I just want to remind everyone, that > > the last community release we had was on 6th of September 2023. So we are > > closing on a year without a release. > > > > Best regards, > > Tibor > > > > Dňa št 18. 7. 2024, 15:28 Alex Porcelli <[email protected]> napísal(a): > > > > > I'll keep the discussion only here (ML) :P > > > > > > @Walter I understand the concern, but I want to highlight that the > > > problem you mentioned will be only fixed when we fix the whole CI > > > system (or restructure the codebase). Until those issues are > > > addressed, it's expected (unfortunately) to have main broken - and > > > this is a good reason why we should start prioritizing the discussions > > > needed to address the root cause of the problems. > > > > > > My point is: we agreed this on the ML, now we are holding what has > > > been agreed because someone is on vacation :( - if this is critical, > > > we - as a community - shouldn't depend on a single individual. > > > > > > With all that being said, it's fine from my side to hold it for 2 > > > weeks, but hopefully not much longer than that. > > > > > > On Thu, Jul 18, 2024 at 9:52 AM Walter Medvedeo <[email protected]> > > > wrote: > > > > > > > > Same here :). > > > > > > > > Alex, > > > > > > > > It must be coordinated because, for example the build of > > > incubator-kie-sonataflow-builder image will disappear from one repo > > > (incubator/kie-tools), and a new build of the same image, will appear, but > > > form a different repo (incubator/incubator-kie-kogito-serverless-operator) > > > > > > > > But, for example incubator-kie-sonataflow-builder, is based on bits from > > > kie-kogito-runtimes. > > > > > > > > If we stop producing incubator-kie-sonataflow-builder from one day to > > > the other, we'll have that image outdated with reference to > > > kie-kogito-runtimes. This represent something no good for the us. > > > (example, > > > if we need to do a fix in kogito-runtimes, we won't have this fix > > > reflected > > > in incubator-kie-sonataflow-builder) > > > > > > > > On the other hand, considering that the CI for > > > incubator-kie-sonataflow-builder is already set, and this image is already > > > being being produced regularly, I don't think this represents a problem > > > nor > > > any extra work. > > > > We just keep all the CI as is, that is already working, and we remove > > > the production for these images "coordinated". > > > > This will help us. > > > > > > > > > > > > On 2024/07/18 12:31:49 Alex Porcelli wrote: > > > > > Now it's my turn to copy my comment :) > > > > > > > > > > I'm not sure what needs to be coordinated... this PR will only > > > > > accomplish what has been decided for a long time. > > > > > > > > > > AFAIK, the original repos from where the content had to be moved never > > > > > stop to be developed... so I can't see why we should hold this agreed > > > > > change. > > > > > > > > > > If the goal is to avoid breaking CI.... then we have a major problem: > > > > > because the CI needs to be fully revisited anyway... and adjust it > > > > > will also take quite some time. > > > > > > > > > > On Thu, Jul 18, 2024 at 3:41 AM Walter Medvedeo <[email protected]> > > > wrote: > > > > > > > > > > > > Hi Alex, thanks for bringing this up. > > > > > > > > > > > > I'll copy here more or less same comment I added in the PR. > > > > > > This removal must be coordinated with the move of this bits to their > > > new destination in apache/incubator-kie-kogito-serverless-operator. We > > > can't remove this as is. > > > > > > > > > > > > We need some time to complete this task, in the mean time we must > > > hold-up the PR. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > On 2024/07/17 22:04:20 Alex Porcelli wrote: > > > > > > > As agreed, here's the link to the PR [1] that removes the Operator > > > and > > > > > > > Images from KIE Tools. > > > > > > > > > > > > > > Of course, as a consequence - once the PR is merged - main won't > > > > > > > be > > > > > > > able to be released anymore. > > > > > > > > > > > > > > [1] - https://github.com/apache/incubator-kie-tools/pull/2472 > > > > > > > > > > > > > > On Tue, Mar 19, 2024 at 12:51 PM Francisco Javier Tirado Sarti > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > Thanks a lot for amending the proposal. > > > > > > > > > > > > > > > > Although I agree with almost the whole proposal, as pointed out > > > in my > > > > > > > > previous e-mails, I did not understand, out of ignorance > > > probably, the need > > > > > > > > for a cutting point of Category A before Category B. If the > > > criteria is the > > > > > > > > existence of a dependency, following that rule, apps should > > > force a cut > > > > > > > > point on runtimes and runtimes should force a cut point on > > > drools. Since we > > > > > > > > are not doing that and everything is still working, I think I > > > should rule > > > > > > > > out that as an explanation. It is because we need to test with > > > certain > > > > > > > > snapshots manually?. Then, the cutting point is almost > > > simultaneous with > > > > > > > > the release isnt it? > > > > > > > > > > > > > > > > If that's the case I propose to slightly amend point 2 by adding > > > manually > > > > > > > > at the end, so eventually, if we manage to test that > > > automatically, then we > > > > > > > > can remove fixed snapshots between repos of category A and B. > > > > > > > > > > > > > > > > 2. Update Category B repos to point to this timestamped > > > SNAPSHOT, and > > > > > > > > verify that everything is working manually. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 18, 2024 at 11:09 PM Alex Porcelli < > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > This is the amended version of the proposal. > > > > > > > > > > > > > > > > > > # THE PROPOSAL > > > > > > > > > > > > > > > > > > S1. [Permanent] We keep the original plan of moving the > > > Quarkus Dev > > > > > > > > > UIs from `kogito-apps` to `kie-tools`, together with > > > Management and > > > > > > > > > Task > > > > > > > > > consoles from `kogito-images` to `kie-tools`. > > > > > > > > > S2. [Temporary] We copy the `kogito-swf-devmode` and > > > > > > > > > `kogito-swf-builder` images from `kogito-images` to > > > `kie-tools` too. > > > > > > > > > S3. [Temporary] We copy the entire > > > `kogito-serverless-operator` repo > > > > > > > > > inside a new package on `kie-tools`. Disable CI for the > > > operator to > > > > > > > > > avoid overlap with kie-tools. > > > > > > > > > > > > > > > > > > > > > > > > > > > List of all repositories that are relevant to Apache KIE > > > 10.0.0 release > > > > > > > > > > > > > > > > > > CATEGORY REPO > > > > > > > > > ===================== > > > > > > > > > A incubator-kie-drools > > > > > > > > > A incubator-kie-optaplanner > > > > > > > > > A incubator-kie-kogito-runtimes > > > > > > > > > A incubator-kie-kogito-apps > > > > > > > > > A incubator-kie-kogito-images > > > > > > > > > ===================== > > > > > > > > > B incubator-kie-sandbox-quarkus-accelerator > > > > > > > > > B incubator-kie-tools > > > > > > > > > > > > > > > > > > * `kogito-serverless-operator` is entirely copied inside > > > `kie-tools`. > > > > > > > > > * `kogito-swf-{builder,devmode}` images also copied to > > > `kie-tools`. > > > > > > > > > * all other repos are ignored for the Apache KIE 10.0.0 > > > > > > > > > release > > > > > > > > > > > > > > > > > > This is the updated steps for the release process itself: > > > > > > > > > > > > > > > > > > 1. Define a timestamped SNAPSHOT to be used as cutting point > > > for > > > > > > > > > Category A repos. > > > > > > > > > 2. Update Category B repos to point to this timestamped > > > SNAPSHOT, and > > > > > > > > > verify that everything is working. > > > > > > > > > 3. At this point, with everything working, we can branch out > > > > > > > > > to > > > > > > > > > `10.0.x`. Category A from the timestamped SNAPSHOT tag, and > > > Category B > > > > > > > > > from `main`. > > > > > > > > > 4. Category A and Category B repos update their versions and > > > > > > > > > dependencies to 10.0.0 in their `10.0.x` branches. > > > > > > > > > 5. Tag 10.0.0-RC1 from `10.0.x`, build source zips and > > > artifacs, and > > > > > > > > > publish to staging. > > > > > > > > > 6. At this point, we can start the vote on the release based > > > on the > > > > > > > > > `10.0.0-RC1` tag. > > > > > > > > > 7. After voting passes, we're good to promote Maven artifact > > > from > > > > > > > > > staging to release. > > > > > > > > > 8. Category A and Category B repos will need a new build to > > > publish > > > > > > > > > the release artifacts, except for Maven artifacts which will > > > > > > > > > be > > > > > > > > > already promoted in the previous step. > > > > > > > > > 9. Once released, remove the temporary code from the > > > `kie-tools` > > > > > > > > > repository on `main` (`kogito-swf-{builder,devmode}` images > > > > > > > > > and > > > > > > > > > `kogito-serverless-operator` codebase). > > > > > > > > > 10. Re-introduce circular dependency in `main` using 10.0.0 > > > fixed > > > > > > > > > versions (in `kie-tools`, `kogito-images` and > > > > > > > > > `kogito-serverless-operator`) to prevent breaking completely > > > CI. > > > > > > > > > Definitive solution must be discussed. > > > > > > > > > > > > > > > > > > Note: The removal of temporary changes must come after the > > > 10.0.0 > > > > > > > > > release, otherwise it'll break the CI for `main`. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 15, 2024 at 1:51 PM Enrique Gonzalez Martinez > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > Parties = two different visions of how to achieve this. > > > > > > > > > > There were two groups of people arguing how to unblock the > > > situation. > > > > > > > > > > > > > > > > > > > > I think this was clear for anyone following this thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > El vie, 15 mar 2024, 18:33, Jason Porter < > > > [email protected]> > > > > > > > > > escribió: > > > > > > > > > > > > > > > > > > > > > On 2024/03/15 12:25:22 Enrique Gonzalez Martinez wrote: > > > > > > > > > > > > +1 to the short term approach as it seems both parties > > > agree into > > > > > > > > > this. > > > > > > > > > > > > > > > > > > > > > > Out of curiosity, what do you mean by "both parties?" > > > Everyone is a > > > > > > > > > > > contributor, there are no companies within the ASF. We're > > > all one big > > > > > > > > > > > community. What we do may not align 100% with the tactical > > > ideas of any > > > > > > > > > > > downstream company, but at the community level, that > > > shouldn't be the > > > > > > > > > > > driver of things anyway. > > > > > > > > > > > > > > > > > > > > > > > Regarding the long term proposal plz move to another > > > thread before > > > > > > > > > > > someone > > > > > > > > > > > > starts engaging in this thread about a topic is not > > > relevant for > > > > > > > > > > > unblocking > > > > > > > > > > > > the version. > > > > > > > > > > > > > > > > > > > > > > > > El vie, 15 mar 2024, 13:20, Kris Verlaenen < > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > escribió: > > > > > > > > > > > > > > > > > > > > > > > > > TLDR: A +1 to the proposal from Tiago for the release, > > > with the > > > > > > > > > > > addition of > > > > > > > > > > > > > some short term recommendation (on how to revert some > > > of the > > > > > > > > > temporary > > > > > > > > > > > > > changes) and some perspective on a potential > > > alternative to > > > > > > > > > consider > > > > > > > > > > > for > > > > > > > > > > > > > the long term > > > > > > > > > > > > > > > > > > > > > > > > > > * Short term: The plan from Tiago describes a strategy > > > that > > > > > > > > > appears to > > > > > > > > > > > be > > > > > > > > > > > > > able to solve the build cycle issues we have, allowing > > > us to > > > > > > > > > proceed > > > > > > > > > > > with > > > > > > > > > > > > > the 10.0 release. We do realize that some of the > > > changes that are > > > > > > > > > > > being > > > > > > > > > > > > > done to be able to do the 10.0 release are going to be > > > temporary. > > > > > > > > > > > > > Therefore, as part of this proposal, I urge the team > > > to also > > > > > > > > > document > > > > > > > > > > > how > > > > > > > > > > > > > we are going to revert some of these temporary changes > > > immediately > > > > > > > > > > > after > > > > > > > > > > > > > the release (*). More specifically, my recommendation > > > is that we > > > > > > > > > agree > > > > > > > > > > > > > that the images and operator folder from kie-tools > > > will be removed > > > > > > > > > > > again > > > > > > > > > > > > > and development will continue on the existing > > > repositories. But > > > > > > > > > let’s > > > > > > > > > > > > > discuss if people see this differently or if there > > > might be other > > > > > > > > > > > steps. > > > > > > > > > > > > > The advantage of this approach would be that it allows > > > us to move > > > > > > > > > > > forward > > > > > > > > > > > > > with the release, does buy us time to find a consensus > > > on the > > > > > > > > > long-term > > > > > > > > > > > > > solution and minimizes the impact on the developers > > > regarding > > > > > > > > > temporary > > > > > > > > > > > > > solutions. And it also requires us to find this > > > consensus before > > > > > > > > > the > > > > > > > > > > > next > > > > > > > > > > > > > release. > > > > > > > > > > > > > > > > > > > > > > > > > > * Longer term: as discussed to some degree in this > > > thread already, > > > > > > > > > > > there > > > > > > > > > > > > > seems to be an alternative to explore where we define > > > more strict > > > > > > > > > > > > > boundaries (for dependencies) between repositories, > > > and create a > > > > > > > > > build > > > > > > > > > > > > > chain where images and operator are built after > > > tools. That said, > > > > > > > > > it’s > > > > > > > > > > > > > fair to say that this proposal needs to be worked out > > > and validated > > > > > > > > > > > more, > > > > > > > > > > > > > and initial assessments on the effort related to this, > > > if we don’t > > > > > > > > > > > want to > > > > > > > > > > > > > rush into this and do things right, are indicating > > > this might take > > > > > > > > > > > multiple > > > > > > > > > > > > > months. We also need to discuss how we will be > > > resourcing this > > > > > > > > > effort. > > > > > > > > > > > > > And we could potentially combine this with other > > > discussions that > > > > > > > > > we > > > > > > > > > > > will > > > > > > > > > > > > > have in the near future. So if we agree to > > > investigate this > > > > > > > > > further, I > > > > > > > > > > > > > would like to recommend moving forward with the more > > > concrete > > > > > > > > > temporary > > > > > > > > > > > > > solution that Tiago is proposing for the 10.0 release. > > > > > > > > > > > > > > > > > > > > > > > > > > Note that this would mean that at this point, on this > > > thread, we > > > > > > > > > don’t > > > > > > > > > > > need > > > > > > > > > > > > > to agree on the specifics of any alternative proposal > > > longer-term, > > > > > > > > > we > > > > > > > > > > > can > > > > > > > > > > > > > start a different conversation thread for this. I > > > hope this can > > > > > > > > > > > convince > > > > > > > > > > > > > people to +1 the approach as described by Tiago short > > > term for the > > > > > > > > > > > release, > > > > > > > > > > > > > with the addition of the recipe how to revert some of > > > the temporary > > > > > > > > > > > changes > > > > > > > > > > > > > and the promise to further evaluate longer-term > > > alternatives. For > > > > > > > > > > > those > > > > > > > > > > > > > that are interested, I wanted to also give an > > > indication what this > > > > > > > > > > > proposal > > > > > > > > > > > > > might mean at a high level from my point of view, > > > which is included > > > > > > > > > > > below. > > > > > > > > > > > > > > > > > > > > > > > > > > Thx, > > > > > > > > > > > > > Kris > > > > > > > > > > > > > > > > > > > > > > > > > > [Optional reading] Alternative longer-term proposal > > > > > > > > > > > > > > > > > > > > > > > > > > One could subdivide the work we do in two main > > > streams: one focused > > > > > > > > > > > more on > > > > > > > > > > > > > the runtimes, one focused more on the tooling. In > > > general a lot of > > > > > > > > > > > tooling > > > > > > > > > > > > > can be built independently from the runtime and vice > > > versa, where > > > > > > > > > they > > > > > > > > > > > > > communicate with each other through well defined > > > formats or apis. > > > > > > > > > > > However, > > > > > > > > > > > > > once we start looking at more advanced use cases and > > > the full > > > > > > > > > > > end-to-end, > > > > > > > > > > > > > this is where we need both tooling and runtime > > > together. > > > > > > > > > > > > > The goal is to create one release pipeline(**). The > > > issue with > > > > > > > > > cyclic > > > > > > > > > > > > > dependencies between repos is imho twofold: 1) we > > > haven’t been 100% > > > > > > > > > > > > > consistent in separating runtimes and tooling this way > > > and 2) we > > > > > > > > > > > haven’t > > > > > > > > > > > > > accommodated well for use cases where runtime and > > > tooling needs to > > > > > > > > > be > > > > > > > > > > > > > combined. Note that some of these dependencies might > > > not be build > > > > > > > > > time > > > > > > > > > > > > > dependencies but test and/or runtime dependencies > > > > > > > > > > > > > only. > > > > > > > > > > > > > As an alternative to one kie-tools monorepo that > > > combines tooling > > > > > > > > > and > > > > > > > > > > > > > images and operator, I believe we can construct a > > > pipeline where > > > > > > > > > most > > > > > > > > > > > of > > > > > > > > > > > > > runtime and tooling can be built independently, but > > > after runtime > > > > > > > > > and > > > > > > > > > > > > > tooling are built, we complete the build with other > > > > > > > > > > > > > components/repositories, because they logically rely > > > more on both. > > > > > > > > > > > > > Examples of components that rely on both are for > > > example be a devui > > > > > > > > > > > > > extension (a quarkus extension that embeds tooling) or > > > the devmode > > > > > > > > > > > image > > > > > > > > > > > > > (that also includes tooling features), or integration > > > testing > > > > > > > > > (where we > > > > > > > > > > > > > want to test whether tooling and runtime work well > > > together. > > > > > > > > > > > > > > > > > > > > > > > > > > More specifically, this would mean > > > > > > > > > > > > > 1) making sure that there are well-defined boundaries > > > between the > > > > > > > > > core > > > > > > > > > > > > > runtimes and core tooling so they don’t depend on each > > > other at > > > > > > > > > build > > > > > > > > > > > > > time. We can decide to move components around where > > > we think that > > > > > > > > > > > makes > > > > > > > > > > > > > sense, for example: > > > > > > > > > > > > > move ui code related to devui into kie-tools (as > > > discussed before) > > > > > > > > > > > > > move kn-workflow to the operator repository as it more > > > closely > > > > > > > > > related > > > > > > > > > > > to > > > > > > > > > > > > > that > > > > > > > > > > > > > 2) update the CI and release pipelines so that core > > > runtime and > > > > > > > > > tooling > > > > > > > > > > > > > repositories can be built first, and are followed up > > > by other > > > > > > > > > > > repositories > > > > > > > > > > > > > like images and operator, that could then rely on > > > > > > > > > > > > > both. > > > > > > > > > > > > > > > > > > > > > > > > > > (*) Note that there would be other options technically > > > to achieve > > > > > > > > > this, > > > > > > > > > > > > > like cutting a release branch early and performing the > > > changes only > > > > > > > > > > > there, > > > > > > > > > > > > > but given other work is still ongoing as well, we want > > > to minimize > > > > > > > > > the > > > > > > > > > > > > > cherry-picking effort. > > > > > > > > > > > > > (**) Note that while the goal is to create one release > > > pipeline, > > > > > > > > > this > > > > > > > > > > > > > should not necessarily mean that we can’t have smaller > > > or optimized > > > > > > > > > > > > > pipelines for CI and daily development, where the > > > impact of > > > > > > > > > changes is > > > > > > > > > > > > > typically more localized. > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 12, 2024 at 8:45 PM Tiago Bento < > > > [email protected] > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unfortunately, I can't do a tl;dr this time, as this > > > matter > > > > > > > > > requires > > > > > > > > > > > a > > > > > > > > > > > > > > lot of context. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This email will take you < 20 minutes to read, > > > according to > > > > > > > > > > > > > > https://thereadtime.com/. > > > > > > > > > > > > > > > > > > > > > > > > > > > > As you may have followed on a separate thread > > > > > > > > > > > > > > ( > > > > > > > > > > > > https://lists.apache.org/thread/nknm6j641qk2c7cl621tsy3fy98tsc69), > > > > > > > > > > > > > > many of us were working towards removing a circular > > > dependency > > > > > > > > > > > > > > currently present between `kogito-apps` and > > > `kie-tools`. As we > > > > > > > > > > > > > > progressed towards a solution, we kept finding the > > > circular > > > > > > > > > > > dependency > > > > > > > > > > > > > > pop up somewhere else. I'll do a breakdown of the > > > things we did, > > > > > > > > > and > > > > > > > > > > > > > > the results we had. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Right now, even though we started the effort to move > > > the Quarkus > > > > > > > > > Dev > > > > > > > > > > > > > > UI modules to `kie-tools`, we haven't been able to > > > do it yet, as > > > > > > > > > > > we've > > > > > > > > > > > > > > been busy upgrading KIE Tools to Java 17, Maven > > > 3.9.6, and > > > > > > > > > Quarkus > > > > > > > > > > > > > > 3.2.9, compatible with Kogito Runtimes > > > 999-20240218-SNAPSHOT. > > > > > > > > > This > > > > > > > > > > > > > > effort was concluded this Monday, with > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/pull/2136. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The current scenario we have is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 01. incubator-kie-kogito-runtimes > > > > > > > > > > > > > > |==> 02. incubator-kie-kogito-apps > > > > > > > > > > > > > > C | 03. incubator-kie-kogito-examples > > > > > > > > > > > > > > Y | 04. incubator-kie-kogito-images > > > > > > > > > > > > > > C | 05. > > > incubator-kie-kogito-serverless-operator > > > > > > > > > > > > > > L | ========================== > > > > > > > > > > > > > > E | 06. > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > > > > > > > > |==> 07. incubator-kie-tools > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * As `kie-tools`/extended-services depends > > > > > > > > > > > > > > on > > > > > > > > > > > > > > `kogito-apps`/jitexecutor; > > > > > > > > > > > > > > * and > > > `kogito-apps`/{sonataflow,bpmn}-quarkus-devui > > > > > > > > > depend on > > > > > > > > > > > > > > `kie-tools`/{many packages} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After moving the Quarkus Dev UIs to `kie-tools`, we > > > would've had: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 01. incubator-kie-kogito-runtimes > > > > > > > > > > > > > > 02. incubator-kie-kogito-apps > > > > > > > > > > > > > > 03. incubator-kie-kogito-examples > > > > > > > > > > > > > > C |==> 04. incubator-kie-kogito-images > > > > > > > > > > > > > > Y | 05. > > > incubator-kie-kogito-serverless-operator > > > > > > > > > > > > > > C | ===================== > > > > > > > > > > > > > > L | 06. > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > > > > > > > > E |==> 07. incubator-kie-tools > > > > > > > > > > > > > > > > > > > > > > > > > > > > * As `kie-tools`/kn-plugin-workflow depends > > > on > > > > > > > > > > > > > > `kogito-images`/kogito-swf-devmode; > > > > > > > > > > > > > > * and `kogito-images`/kogito-swf-devmode > > > depends on > > > > > > > > > > > > > > `kie-tools`/sonataflow-quarkus-devui > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After moving the `kogito-swf-devmode` image to > > > `kie-tools`, we > > > > > > > > > > > would've > > > > > > > > > > > > > > had: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 01. incubator-kie-kogito-runtimes > > > > > > > > > > > > > > 02. incubator-kie-kogito-apps > > > > > > > > > > > > > > 03. incubator-kie-kogito-examples > > > > > > > > > > > > > > 04. incubator-kie-kogito-images > > > > > > > > > > > > > > C |==> 05. > > > incubator-kie-kogito-serverless-operator > > > > > > > > > > > > > > Y | ===================== > > > > > > > > > > > > > > C | 06. > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > > > > > > > > L |==> 07. incubator-kie-tools > > > > > > > > > > > > > > E > > > > > > > > > > > > > > > > > > > > > > > > > > > > * As `kie-tools`/kn-plugin-workflow depends > > > on > > > > > > > > > > > > > > `kogito-serverless-operator`; > > > > > > > > > > > > > > * and `kogito-serverless-operator` depends > > > > > > > > > > > > > > on > > > > > > > > > > > > > > `kie-tools`/kogito-swf-devmode > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Clearly, we have a much bigger problem than a simple > > > circular > > > > > > > > > > > dependency. > > > > > > > > > > > > > > > > > > > > > > > > > > > > After multiple conversations with a lot of people, > > > it's been > > > > > > > > > really > > > > > > > > > > > > > > hard coming up with a simple solution that makes it > > > possible to > > > > > > > > > build > > > > > > > > > > > > > > Apache KIE in one shot, while preserving the way > > > everyone is > > > > > > > > > used to > > > > > > > > > > > > > > contributing to the multiple repositories we have. > > > More than > > > > > > > > > that, > > > > > > > > > > > > > > while making this assessment, I found more problems > > > that, in my > > > > > > > > > > > > > > perspective, block Apache KIE 10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > In light of that difficulty, I'm coming forward with > > > my proposal > > > > > > > > > for > > > > > > > > > > > > > > the Apache KIE release process, so we can use > > > Apache's > > > > > > > > > mechanisms to > > > > > > > > > > > > > > have a slower-paced, in-depth debate about this > > > really > > > > > > > > > complicated > > > > > > > > > > > > > > matter. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll lay out my entire perspective about the current > > > situation > > > > > > > > > of our > > > > > > > > > > > > > > codebase, as well as problems I can currently see. > > > I'll start > > > > > > > > > with an > > > > > > > > > > > > > > analysis of the repositories and their purposes, > > > point out some > > > > > > > > > > > > > > problems that I believe are blocking our 10 release, > > > explain my > > > > > > > > > > > > > > proposal and discuss some consequences to what I'm > > > proposing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let's begin. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > # THE APACHE KIE REPOS > > > > > > > > > > > > > > > > > > > > > > > > > > > > A. DROOLS OPTAPLANNER, & KOGITO (count: 11) > > > > > > > > > > > > > > - incubator-kie-kogito-pipelines @ `main` > > > > > > > > > > > > > > - incubator-kie-drools @ `main` > > > > > > > > > > > > > > - incubator-kie-optaplanner @ `main` > > > > > > > > > > > > > > - incubator-kie-optaplanner-quickstarts @ `main` > > > > > > > > > > > > > > - incubator-kie-kogito-runtimes @ `main` > > > > > > > > > > > > > > - incubator-kie-kogito-apps @ `main` > > > > > > > > > > > > > > - incubator-kie-kogito-examples @ `main` > > > > > > > > > > > > > > - incubator-kie-kogito-images @ `main` > > > > > > > > > > > > > > - incubator-kie-kogito-serverless-operator @ `main` > > > > > > > > > > > > > > - incubator-kie-kogito-docs @ `main` > > > > > > > > > > > > > > - incubator-kie-docs @ `main-kogito` > > > > > > > > > > > > > > > > > > > > > > > > > > > > B. TOOLS (count: 2) > > > > > > > > > > > > > > - incubator-kie-sandbox-quarkus-accelerator @ > > > > > > > > > > > > > > `0.0.0` > > > > > > > > > > > > > > - incubator-kie-tools @ `main` > > > > > > > > > > > > > > > > > > > > > > > > > > > > C. BENCHMARKS (count: 2) > > > > > > > > > > > > > > - incubator-kie-kogito-benchmarks @ `main` > > > > > > > > > > > > > > - incubator-kie-benchmarks @ `main` > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. ARCHIVED (count: 1) > > > > > > > > > > > > > > - incubator-kie-kogito-operator > > > > > > > > > > > > > > > > > > > > > > > > > > > > E. "NON-CODE" (count: 5) > > > > > > > > > > > > > > - incubator-kie-issues @ `main` > > > > > > > > > > > > > > (Issues only, README should be updated @ `main`. > > > Same for > > > > > > > > > GitHub > > > > > > > > > > > > > > Actions workflows.) > > > > > > > > > > > > > > - incubator-kie-kogito-website @ `main` > > > > > > > > > > > > > > (The Kogito website. Develop & deploy at the > > > `main` branch.) > > > > > > > > > > > > > > - incubator-kie-website @ `main` > > > > > > > > > > > > > > (The KIE website. Develop @ `main`. Push @ > > > `deploy` to > > > > > > > > > update the > > > > > > > > > > > > > > website.) > > > > > > > > > > > > > > - incubator-kie-kogito-online @ `gh-pages` > > > > > > > > > > > > > > (GitHub pages used to host sandbox.kie.org and > > > KIE Tools' > > > > > > > > > Chrome > > > > > > > > > > > > > > Extension assets.) > > > > > > > > > > > > > > - incubator-kie-kogito-online-staging @ `main` > > > > > > > > > > > > > > (Same as above, but for manual sanity checks > > > during the > > > > > > > > > staging > > > > > > > > > > > > > > phase of a release.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > TOTAL (count: 21) > > > > > > > > > > > > > > > > > > > > > > > > > > > > I grouped the repositories by category, and listed > > > them in a > > > > > > > > > > > > > > topological order. Keep in mind that when flattening > > > out a tree, > > > > > > > > > > > there > > > > > > > > > > > > > > are multiple possibilities. For example, OptaPlanner > > > could've > > > > > > > > > been > > > > > > > > > > > > > > placed in any position after Drools. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Category A repos are what I've been referring to as > > > `drools` and > > > > > > > > > > > > > > `kogito-*` stream. Of course OptaPlanner is inside > > > that stream, > > > > > > > > > as > > > > > > > > > > > the > > > > > > > > > > > > > > way these repositories reference each other are > > > through Maven > > > > > > > > > > > > > > SNAPSHOTs. More specifically, the 999-SNAPSHOT > > > version. This > > > > > > > > > > > mechanism > > > > > > > > > > > > > > is well-known to the team, and although flawed for > > > intra-day > > > > > > > > > builds > > > > > > > > > > > > > > and disruptive for people in many different time > > > zones, it is > > > > > > > > > already > > > > > > > > > > > > > > very comfortable for everyone to work with, I > > > > > > > > > > > > > > assume. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Contributions made to Category A have some dedicated > > > pipelines, > > > > > > > > > which > > > > > > > > > > > > > > are, at least to some extent, able to build > > > cross-repo PRs > > > > > > > > > together > > > > > > > > > > > > > > and verify that the codebase will continue working > > > as expected > > > > > > > > > after > > > > > > > > > > > > > > they're all merged. From what I could gather, there > > > are some > > > > > > > > > > > > > > "sub-streams" currently configured for cross-repo > > > PRs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > - kogito-pipelines > > > > > > > > > > > > > > - drools, kogito-runtimes, kogito-apps, and > > > kogito-examples > > > > > > > > > > > > > > - optaplanner, and optaplanner-quickstarts > > > > > > > > > > > > > > - kogito-images, and kogito-serverless-operator > > > > > > > > > > > > > > - kogito-docs > > > > > > > > > > > > > > - kie-docs > > > > > > > > > > > > > > > > > > > > > > > > > > > > This means that sending cross-repo PRs to any > > > combination of > > > > > > > > > repos > > > > > > > > > > > > > > that are not part of the same "sub-stream" cannot be > > > verified > > > > > > > > > before > > > > > > > > > > > > > > merging, making our contribution model dependent on > > > individual > > > > > > > > > > > > > > contributors building stuff on their machines to > > > verify that it > > > > > > > > > > > works. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I based this analysis on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/project-dependencies.yaml > > > > > > > > > > > > > > , > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-optaplanner/blob/main/.ci/buildchain-project-dependencies.yaml > > > > > > > > > > > > > > , > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/jenkins/config/branch.yaml > > > > > > > > > > > > > > . > > > > > > > > > > > > > > Note that I'm not that familiar with these > > > pipelines, so please > > > > > > > > > > > > > > someone correct me if I'm wrong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Category B repos are what I've been referring to as > > > `kie-tools` > > > > > > > > > > > > > > stream. The first repo there is a template > > > repository that is > > > > > > > > > used by > > > > > > > > > > > > > > people starting projects from scratch on KIE > > > Sandbox, similar to > > > > > > > > > a > > > > > > > > > > > > > > Maven archetype, if you will. The other one is the > > > KIE Tools > > > > > > > > > > > monorepo, > > > > > > > > > > > > > > a polyglot monorepo with `pnpm` as its build system. > > > Currently, > > > > > > > > > KIE > > > > > > > > > > > > > > Tools hosts Java libraries and apps, TypeScript > > > libraries and > > > > > > > > > apps, > > > > > > > > > > > Go > > > > > > > > > > > > > > apps, Docker images, and Helm charts. The > > > `kie-tools` monorepo is > > > > > > > > > > > > > > configured to work with sparse checkouts and can do > > > partial > > > > > > > > > builds. > > > > > > > > > > > > > > Category B repos refer to Category A repos through > > > timestamped > > > > > > > > > > > > > > SNAPSHOTs. This is a new mechanism we recently > > > introduced that > > > > > > > > > will > > > > > > > > > > > > > > build and publish immutable, persistent artifacts > > > under a version > > > > > > > > > > > > > > following the 999-YYYYMMDD-SNAPSHOT format, > > > published weekly > > > > > > > > > every > > > > > > > > > > > > > > Sunday night. Timestamped SNAPSHOTs are an evolution > > > to the > > > > > > > > > Kogito > > > > > > > > > > > > > > releases, as we're now targeting one release for all > > > of Apache > > > > > > > > > KIE, > > > > > > > > > > > so > > > > > > > > > > > > > > we can't have Kogito releases anymore. > > > > > > > > > > > > > > > > > > > > > > > > > > > > An important note here is that Category B > > > repositories have been > > > > > > > > > > > > > > historically kept out of any automations we used to > > > have, way > > > > > > > > > back > > > > > > > > > > > > > > when Kogito started and we had the Business Central > > > (a.k.a. v7) > > > > > > > > > > > stream > > > > > > > > > > > > > > still going on. For this reason, Category B projects > > > have > > > > > > > > > developed > > > > > > > > > > > > > > their own automations, based on GitHub Actions. > > > Category B repos > > > > > > > > > have > > > > > > > > > > > > > > always depended on Category A repos using fixed > > > versions. If > > > > > > > > > Category > > > > > > > > > > > > > > B repos have had adopted mutable SNAPSHOTs, breaking > > > changes on > > > > > > > > > > > > > > Category A repositories would've had the potential > > > to break > > > > > > > > > Category > > > > > > > > > > > B > > > > > > > > > > > > > > silently, leaving Category B with a broken > > > development stream, > > > > > > > > > and > > > > > > > > > > > > > > introducing unpleasant surprises for maintainers of > > > Category B > > > > > > > > > repos, > > > > > > > > > > > > > > as historically Category A contributors were not > > > familiar with > > > > > > > > > > > > > > Category B repos. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Contributions made to Category B repos go through a > > > GitHub > > > > > > > > > Actions > > > > > > > > > > > > > > workflow that builds the relevant part of the > > > `kie-tools` > > > > > > > > > monorepo > > > > > > > > > > > for > > > > > > > > > > > > > > the changes introduced. Changes made to the pipeline > > > itself are > > > > > > > > > also > > > > > > > > > > > > > > picked up as part of PRs, allowing us to do things > > > like > > > > > > > > > atomically > > > > > > > > > > > > > > bumping the Node.js version, for example. More > > > importantly, it > > > > > > > > > allows > > > > > > > > > > > > > > us to upgrade the repository to a new timestamped > > > SNAPSHOT > > > > > > > > > together > > > > > > > > > > > > > > with the changes necessary to make it stay green. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This setup, however, makes it impossible to have > > > cross-repo PRs > > > > > > > > > > > > > > involving Category A and Category B simultaneously, > > > with the > > > > > > > > > current > > > > > > > > > > > > > > automations we have. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Category C repos are kind of floating around, and > > > I'm not sure if > > > > > > > > > > > > > > there's much activity going on there. Regardless, as > > > they're > > > > > > > > > part of > > > > > > > > > > > > > > Apache KIE, they will be part of our release, so I > > > listed them > > > > > > > > > for us > > > > > > > > > > > > > > to take them into consideration too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Category D is self explanatory. There's only one > > > repo that has > > > > > > > > > > > already > > > > > > > > > > > > > > been marked for being archived. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Category E are repos that do not host code directly, > > > and are > > > > > > > > > either > > > > > > > > > > > > > > organizational entities, or host websites, that > > > currently are not > > > > > > > > > > > part > > > > > > > > > > > > > > of any pipelines we have. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This lack of unification between Category A and > > > Category B is, > > > > > > > > > IMHO, > > > > > > > > > > > > > > what allowed us to introduce the infamous circular > > > dependency > > > > > > > > > between > > > > > > > > > > > > > > `kie-tools` and `kogito-apps`, which we now can > > > describe as a > > > > > > > > > > > circular > > > > > > > > > > > > > > dependency between Category A and Category B. The > > > way I see it, > > > > > > > > > if we > > > > > > > > > > > > > > had a single pipeline, building everything from > > > `drools` to > > > > > > > > > > > > > > `kie-tools`, such flaws would've never been > > > introduced, and we > > > > > > > > > > > > > > wouldn't be having this huge problem in our hands > > > right now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > My proposal for the Apache KIE release process sees > > > this lack of > > > > > > > > > > > > > > unification as a central problem, not only for this > > > release in > > > > > > > > > > > > > > particular, but for the community as a whole. It is > > > my belief > > > > > > > > > that we > > > > > > > > > > > > > > are all under the same roof, and that no > > > contribution should be > > > > > > > > > > > > > > allowed to break any part of our codebase. With the > > > increasing > > > > > > > > > volume > > > > > > > > > > > > > > of code, and hopefully number of contributors too, > > > we cannot keep > > > > > > > > > > > > > > counting on "common sense" to avoid breaking things. > > > We're all > > > > > > > > > humans > > > > > > > > > > > > > > after all, and it is our job to have mechanisms in > > > place to > > > > > > > > > prevent > > > > > > > > > > > us > > > > > > > > > > > > > > from unwillingly making mistakes. Especially when > > > these mistakes > > > > > > > > > > > > > > impact on parts of the codebase that we, > > > individually, probably > > > > > > > > > can't > > > > > > > > > > > > > > fix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > # THE PROBLEMS WE HAVE RIGHT NOW > > > > > > > > > > > > > > > > > > > > > > > > > > > > P1. Quarkus Dev UIs @ `kogito-apps` depending on > > > kiegroup's KIE > > > > > > > > > Tools > > > > > > > > > > > > > > `0.32.0`. > > > > > > > > > > > > > > See: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/search?q=repo%3Akiegroup%2Fkogito-apps+path%3Apackage.json+kie-tools&type=code > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > P2. PR open for Kogito SWF images @ `kogito-images` > > > depending on > > > > > > > > > > > > > > kiegroup's KIE Tools `0.32.0`. > > > > > > > > > > > > > > See: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/tree/main/packages/sonataflow-deployment-webapp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > P3. DashBuilder @ `kie-tools` depending on > > > kiegroup's `lienzo` > > > > > > > > > and > > > > > > > > > > > > > > `kie-soup` artifacts at version `7.59.0.Final`. > > > > > > > > > > > > > > See: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/blob/main/packages/dashbuilder/pom.xml#L64 > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/search?q=repo%3Aapache%2Fincubator-kie-tools+path%3Apackages%2Fdashbuilder+%24%7Bversion.org.kie%7D&type=code > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > P4. Multiple packages @ `kogito-apps` depending on > > > kiegroup's > > > > > > > > > > > > > > Explainability `1.22.1.Final`. > > > > > > > > > > > > > > * This module was removed from the KIE codebase > > > > > > > > > > > > > > here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-apps/commit/bbb22c06d37e77b97aae6496d74abe43a8cfc965 > > > > > > > > > > > > > > and now lives on > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/trustyai-explainability/trustyai-explainability, > > > > > > > > > > > > > > under a different GAV. > > > > > > > > > > > > > > * This new repo depends on Kogito and OptaPlanner, > > > pointing to > > > > > > > > > older > > > > > > > > > > > > > > versions. > > > > > > > > > > > > > > See: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/search?q=repo%3Aapache%2Fincubator-kie-kogito-apps+%3Eexplainability-core%3C&type=code > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/trustyai-explainability/trustyai-explainability/blob/main/pom.xml#L52-L53 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > P5. `incubator-kie-sandbox-quarkus-accelerator` > > > depending on > > > > > > > > > Kogito > > > > > > > > > > > > > > `1.32.0.Final` and Quarkus `2.15.3.Final`. > > > > > > > > > > > > > > See: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-sandbox-quarkus-accelerator/blob/0.0.0/pom.xml#L32-L38 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > P6. Category C repos are out of date and not part of > > > the > > > > > > > > > Category A > > > > > > > > > > > > > > CI/Release pipelines. > > > > > > > > > > > > > > * incubator-kie-kogito-benchmarks: (Current version > > > is > > > > > > > > > > > `2.0-SNAPSHOT`, > > > > > > > > > > > > > > depending on Kogito without a specific version, only > > > by using > > > > > > > > > > > > > > `http://localhost:8080`) > > > > > > > > > > > > > > * incubator-kie-benchmarks: (Current version is > > > `1.0-SNAPSHOT`, > > > > > > > > > > > > > > pointing to Drools 999-SNAPSHOT and OptaPlanner > > > > > > > > > `8.45.0-SNAPSHOT`) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > P7. `kie-tools`/packages/kn-plugin-workflow has its > > > E2E disabled > > > > > > > > > > > after > > > > > > > > > > > > > > upgrading to 999-20240218-SNAPSHOT. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In my perspective, P1 and P2 have the same solution, > > > as they both > > > > > > > > > > > > > > suffer from the circular dependency between Category > > > A and > > > > > > > > > Category > > > > > > > > > > > B. > > > > > > > > > > > > > > As Category A and Category B are both streams that > > > have been > > > > > > > > > really > > > > > > > > > > > > > > active, I see this as a blocker, as there are > > > contributions that > > > > > > > > > > > > > > cannot be done, given that Category A depends on > > > Category B with > > > > > > > > > a > > > > > > > > > > > > > > dephasing of 1 release. > > > > > > > > > > > > > > > > > > > > > > > > > > > > P3 and P4, although not ideal, can be understood as > > > technical > > > > > > > > > debt. > > > > > > > > > > > > > > Depending on unmaintained projects is something > > > we'll always be > > > > > > > > > > > > > > susceptible to, given time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > P5 and P6 are easily fixable, as it's just a matter > > > of making > > > > > > > > > them > > > > > > > > > > > > > > part of the play. > > > > > > > > > > > > > > > > > > > > > > > > > > > > P7 is an isolated problem that won't impact the > > > structure or > > > > > > > > > anything > > > > > > > > > > > > > > that we're talking about here, but it is a > > > regression we > > > > > > > > > introduced > > > > > > > > > > > > > > recently. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assuming P3 and P4 can be ignored for Apache KIE 10, > > > and that > > > > > > > > > P5, P6, > > > > > > > > > > > > > > and P7 have easy fixes, the only problems left to > > > discuss are P1 > > > > > > > > > and > > > > > > > > > > > > > > P2, which can't be done without a proper proposal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > # THE PROPOSAL > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll try to be very meticulous here, since from my > > > experience, > > > > > > > > > any > > > > > > > > > > > > > > little miscalculation can lead to our release not > > > working out in > > > > > > > > > the > > > > > > > > > > > > > > end. To try and avoid that as much as possible, and > > > make > > > > > > > > > everything > > > > > > > > > > > we > > > > > > > > > > > > > > can to have a successful Apache KIE 10 release, bear > > > with me. > > > > > > > > > I'll > > > > > > > > > > > lay > > > > > > > > > > > > > > out a timeline of events that need to happen in > > > order for our > > > > > > > > > release > > > > > > > > > > > > > > to be published, with all artifacts ending up in the > > > right > > > > > > > > > places, > > > > > > > > > > > but > > > > > > > > > > > > > > first, we need to solve problems P1 and P2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > As you saw at the beginning of this email, all the > > > attempts we > > > > > > > > > made > > > > > > > > > > > > > > left us with the circular dependency showing up at a > > > different > > > > > > > > > place, > > > > > > > > > > > > > > but something all these places have in common is > > > that they're all > > > > > > > > > > > > > > after kogito-apps, and before to Category B. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first part of my proposal is the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > S1. We keep the original plan of moving the Quarkus > > > Dev UIs from > > > > > > > > > > > > > > `kogito-apps` to `kie-tools`, together with > > > Management and Task > > > > > > > > > > > > > > consoles from `kogito-images` to `kie-tools`. > > > > > > > > > > > > > > S2. We move the `kogito-swf-devmode` and > > > `kogito-swf-builder` > > > > > > > > > images > > > > > > > > > > > > > > from `kogito-images` to `kie-tools` too. > > > > > > > > > > > > > > S3. We move the entire `kogito-serverless-operator` > > > repo inside > > > > > > > > > a new > > > > > > > > > > > > > > package on `kie-tools`, keeping Git history. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Solutions S1, S2, and S3 together solve problems P1 > > > and P2. Of > > > > > > > > > course > > > > > > > > > > > > > > the rest of > > > > > > > > > > > https://github.com/apache/incubator-kie-issues/issues/967 > > > > > > > > > > > > > > would still be done too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This doesn't come without consequences, of course, > > > as the > > > > > > > > > > > > > > `kogito-swf-devmode` and `kogito-swf-builder` > > > images, and the > > > > > > > > > > > > > > `kogito-serverless-operator` would be moving from > > > Category A to > > > > > > > > > > > > > > Category B. This move would make them have to > > > reference Category > > > > > > > > > A > > > > > > > > > > > > > > repos through timestamped SNAPSHOTs. Since > > > `kogito-images` and > > > > > > > > > > > > > > `kogito-serverless-operator` are already their own > > > "sub-stream" > > > > > > > > > > > inside > > > > > > > > > > > > > > Category A, though, contributions made in a > > > cross-repo fashion to > > > > > > > > > > > this > > > > > > > > > > > > > > "sub-stream" will continue being possible, now via a > > > single PR to > > > > > > > > > > > > > > `kie-tools`. Cross-repo PRs between Category A and > > > Category B > > > > > > > > > will > > > > > > > > > > > > > > continue not being possible, and a 1-week delay > > > between merging > > > > > > > > > > > > > > something on Category A and using it on Category B > > > will still > > > > > > > > > happen. > > > > > > > > > > > > > > > > > > > > > > > > > > > > It's worth mentioning that `kie-tools`, however, > > > does allow for > > > > > > > > > > > sparse > > > > > > > > > > > > > > checkouts and partial builds, so working with a > > > subset of the > > > > > > > > > > > monorepo > > > > > > > > > > > > > > is possible and encouraged. Making changes only to > > > > > > > > > > > > > > `packages/kn-plugin-workflow`, for example, will > > > have the PR > > > > > > > > > checks > > > > > > > > > > > > > > run in < 10 minutes, as you can see here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-tools/actions/runs/8237244382/job/22525511722?pr=2136 > > > > > > > > > > > > > > . > > > > > > > > > > > > > > We're not compromising when running partial builds > > > too. We know > > > > > > > > > that > > > > > > > > > > > > > > the entire repo will continue working even after > > > only building a > > > > > > > > > > > small > > > > > > > > > > > > > > subset of the changes. Doing partial or full builds > > > is > > > > > > > > > automatically > > > > > > > > > > > > > > determined by the changes of a PR. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keep in mind that, even though I'm proposing we move > > > a bunch of > > > > > > > > > > > > > > additional stuff into `kie-tools`, I see this as a > > > TEMPORARY > > > > > > > > > solution > > > > > > > > > > > > > > for our codebase. `kie-tools` would host some > > > additional stuff > > > > > > > > > > > > > > TEMPORARILY so that we can release and continue > > > moving forward. > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned on other places, `kie-tools` became a > > > polyglot > > > > > > > > > > > monorepo > > > > > > > > > > > > > > out of necessity, and although I'm really proud of > > > what we > > > > > > > > > achieved > > > > > > > > > > > > > > there so far, I don't think `kie-tools` has a setup > > > that is > > > > > > > > > suitable > > > > > > > > > > > > > > for all the different nuances that compose our > > > community. I'm > > > > > > > > > well > > > > > > > > > > > > > > aware that a polyglot monorepo that does not follow > > > widespread > > > > > > > > > > > > > > conventions will scare some people away, and as much > > > as we've > > > > > > > > > tried > > > > > > > > > > > to > > > > > > > > > > > > > > make build instructions clear, we can't always get > > > past the > > > > > > > > > prejudice > > > > > > > > > > > > > > some people have towards the "front-end" ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > With all that said, I keep thinking this is the best > > > course of > > > > > > > > > action > > > > > > > > > > > > > > for us right now. We keep most of our stuff > > > unchanged, we > > > > > > > > > unblock the > > > > > > > > > > > > > > release, and we have a working setup that will suit > > > us well > > > > > > > > > while we > > > > > > > > > > > > > > discuss and reach a conclusion regarding the future > > > of our > > > > > > > > > codebase > > > > > > > > > > > > > > structure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me paint a quick picture here of what our code > > > base would > > > > > > > > > look > > > > > > > > > > > > > > like, repository-wise, if my proposal is accepted: > > > > > > > > > > > > > > > > > > > > > > > > > > > > CATEGORY REPO > > > > > > > > > > > > > > ===================== > > > > > > > > > > > > > > A incubator-kie-kogito-pipelines > > > > > > > > > > > > > > A incubator-kie-drools > > > > > > > > > > > > > > A incubator-kie-optaplanner > > > > > > > > > > > > > > A incubator-kie-optaplanner-quickstarts > > > > > > > > > > > > > > A incubator-kie-kogito-runtimes > > > > > > > > > > > > > > A incubator-kie-kogito-apps > > > > > > > > > > > > > > A incubator-kie-kogito-examples > > > > > > > > > > > > > > A incubator-kie-kogito-images > > > > > > > > > > > > > > A incubator-kie-kogito-docs > > > > > > > > > > > > > > A incubator-kie-kogito-benchmarks > > > > > > > > > > > > > > A incubator-kie-docs > > > > > > > > > > > > > > A incubator-kie-benchmarks > > > > > > > > > > > > > > ===================== > > > > > > > > > > > > > > B > > > > > > > > > > > > > > incubator-kie-sandbox-quarkus-accelerator > > > > > > > > > > > > > > B incubator-kie-tools > > > > > > > > > > > > > > ===================== > > > > > > > > > > > > > > D incubator-kie-kogito-operator > > > > > > > > > > > > > > ===================== > > > > > > > > > > > > > > E incubator-kie-issues > > > > > > > > > > > > > > E incubator-kie-kogito-website > > > > > > > > > > > > > > E incubator-kie-website > > > > > > > > > > > > > > E incubator-kie-kogito-online > > > > > > > > > > > > > > E incubator-kie-kogito-online-staging > > > > > > > > > > > > > > ===================== > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Category C becomes part of Category A, and > > > > > > > > > > > > > > `kogito-serverless-operator` moves entirely inside > > > `kie-tools`. > > > > > > > > > > > > > > * With `kogito-swf-{builder,devmode}` images and > > > > > > > > > > > > > > `kogito-serverless-operator` inside `kie-tools`, > > > there are no > > > > > > > > > cycles > > > > > > > > > > > > > > anymore, as inside `kie-tools`, we can granularly > > > build: > > > > > > > > > > > > > > 1. packages/sonataflow-deployment-webapp > > > > > > > > > > > > > > 2. packages/sonataflow-quarkus-devui > > > > > > > > > > > > > > 3. packages/sonataflow-images (containing > > > > > > > > > `kogito-swf-builder` > > > > > > > > > > > and > > > > > > > > > > > > > > `kogito-swf-devmode`) > > > > > > > > > > > > > > 4. packages/sonataflow-operator (contents from > > > > > > > > > > > > > > `kogito-serverless-operator`) > > > > > > > > > > > > > > 5. packages/kn-plugin-sonataflow > > > > > > > > > (`packages/kn-plugin-workflow`, > > > > > > > > > > > > > > but renamed) > > > > > > > > > > > > > > > > > > > > > > > > > > > > The second part of the proposal is the release > > > process itself, > > > > > > > > > > > > > > assuming the structure above is what we have. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here it is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Define a timestamped SNAPSHOT to be used as > > > cutting point for > > > > > > > > > > > > > > Category A repos. > > > > > > > > > > > > > > 2. Update Category B repos to point to this > > > timestamped > > > > > > > > > SNAPSHOT, and > > > > > > > > > > > > > > verify that everything is working. > > > > > > > > > > > > > > 3. At this point, with everything working, we can > > > branch out to > > > > > > > > > > > > > > `10.0.x`. Category A from the timestamped SNAPSHOT > > > tag, and > > > > > > > > > Category > > > > > > > > > > > B > > > > > > > > > > > > > > from `main`. > > > > > > > > > > > > > > 4. All Category A and Category B repos update their > > > versions to > > > > > > > > > > > > > > 10.0.0, in their `10.0.x` branches. > > > > > > > > > > > > > > 5. Update Category B repos to point to Category A > > > repos using the > > > > > > > > > > > > > > 10.0.0 version. > > > > > > > > > > > > > > 6. At this point, we can vote on the release based > > > on the > > > > > > > > > `10.0.x` > > > > > > > > > > > > > > branches, given we don't expect any code changes > > > anymore. > > > > > > > > > > > > > > 7. After voting passes, we're good to start the > > > release process. > > > > > > > > > > > > > > 8. Category A repos follow their manual/automated > > > release > > > > > > > > > process, > > > > > > > > > > > > > > pointing to the `10.0.x` branch. Tags pushed to Git, > > > and built > > > > > > > > > > > > > > artifacts pushed to their registries. > > > > > > > > > > > > > > 9. We wait a little bit for Category A artifacts to > > > be > > > > > > > > > propagated on > > > > > > > > > > > > > > registries. ~1 day. > > > > > > > > > > > > > > 10. Category B repos follow their manual/automated > > > release > > > > > > > > > process, > > > > > > > > > > > > > > pointing to the `10.0.x` branch. Tags pushed to Git, > > > and built > > > > > > > > > > > > > > artifacts pushed to their registries. > > > > > > > > > > > > > > 11. Category D repos are ignored. > > > > > > > > > > > > > > 12. Category E repos can be manually tagged with > > > 10.0.0 from > > > > > > > > > their > > > > > > > > > > > > > > default branches. > > > > > > > > > > > > > > > > > > > > > > > > > > > > More needs to be discussed if we're planning to > > > maintain multiple > > > > > > > > > > > > > > release streams in parallel, but I guess it can wait > > > for after > > > > > > > > > Apache > > > > > > > > > > > > > > KIE 10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for reading, and I'm looking forward to > > > hearing back > > > > > > > > > from > > > > > > > > > > > > > > everyone. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Of course, alternative solutions are possible. This > > > email, > > > > > > > > > however, > > > > > > > > > > > > > > summarizes my view of how we should attack the > > > problem, > > > > > > > > > considering > > > > > > > > > > > > > > disruption, required effort, the release process > > > itself, and > > > > > > > > > history. > > > > > > > > > > > > > > Feel free to propose alternatives. This is not a > > > voting thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tiago Bento > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > [email protected] > > > > > > > > > > > > > > For additional commands, e-mail: > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
