Hi Enrique! The images are using SNAPSHOTS from apps/runtimes currently. There was a decision in the past to build apps in the images from the main/versioned branch. We are changing this to use SNAPSHOT/versioned artifacts published in Maven. About kie-tools, I don't know either.
We can use immutable snapshots (weekly, nightlies) instead. I'm not against it. But we do need those for integration tests or at least a robust CI platform to do integration without snapshots. Cheers! On Thu, May 9, 2024 at 5:07 AM Enrique Gonzalez Martinez < [email protected]> wrote: > Hi Tiago, > > I think the discussion is related to the graph dependency for now. So > I would keep that in mind. Anyway I am kinda surprised about some > concepts like (mutable snapshots and using snapshot published > timestamps like they are final). > > Now I think I understand where the problem lies now; so basically if > we had the proper graph dependency project built based on proper > snapshots everything would build just ok. So my question is pretty > simple, why is it not possible for kie tools or kogito images to use > the maven snapshot concept and point out to the last snapshot build > (either local or remote) like other maven projects work. Is there any > limitation there ? > > El mié, 8 may 2024 a las 22:30, Tiago Bento (<[email protected]>) > escribió: > > > > Hi Zanini! Welcome back, hope you had a great time during your PTO. > > > > If I understand correctly, the way you see the integration between > > repos relies heavily on timestamped SNAPSHOTs. This way: > > 1. `kogito-images` would stop building `kogito-apps`, and would rely > > on timestamped SNAPSHOTs being published from > > `drools/optaplanner/kogito-runtimes/kogito-apps`. > > 2. `kie-tools` would start publishing timestamped SNAPSHOTs for its UI > > components, so they would be consumed downstream by > > `kogito-serverless-operator`. > > > > If that's what you mean, let me elaborate on some of the consequences > > of this integration model, where we stop using the mutable > > 999-SNAPSHOT and start relying more on timestamped SNAPSHOTs for > > cross-repo synchronization: > > a. Cross-repo PRs would not be possible anymore, depending on which > > repos are involved in the ensemble. This would create boundaries > > between certain groups of our repos, namely: > > drools/optaplanner/kogito-runtimes/kogito-apps --> kogito-images --> > > kie-tools, --> kogito-serverless-operator. So 4 "independent > > clusters". > > b. kogito-images, kie-tools, and kogito-serverless-operator, using > > timestamped SNAPSHOTs between themselves, would make the most upstream > > of the three (in this case, `kogito-images`), to be the one defining > > what timestamped SNAPSHOT version we should use for its upstream > > dependencies (drools/optaplanner/kogito-runtimes/kogito-apps). In more > > details: If `kogito-images` depends on > > `drools/optaplanner/kogito-runtimes/kogito-apps` using version > > 999-20240501, that's the version `kie-tools` and > > `kogito-serverless-operator` need to use as well, as the > > drools/optaplanner/kogito-runtimes/kogito-apps would come transitively > > from the `kogito-images` dependency these two repos would have. This, > > IMHO, causes a lot of confusion, since the dates on the timestamped > > SNAPSHOT versions have implicit transitive dependencies to other > > upstream projects that also have their own timestamped SNAPSHOTs > > published. I.e., 999-20240501 of `kie-tools` won't be aligned with > > 999-2024051 of `drools/optaplanner/kogito-runtimes/kogito-apps`. > > c. Updating dependencies across the board would be much more > > difficult, as we would need to rely on a chain reaction starting from > > the `drools/optaplanner/kogito-runtimes/kogito-apps` cluster all the > > way down to `kogito-serverless-operator`, releasing timestamped > > SNAPSHOTs and sending PRs for each "cluster" along the way. > > > > I'm not sure this is the way we want to move forward, to be honest, as > > these consequences need to be thoroughly examined by people in the > > community, so that all the side-effects of this choice are clear to > > everyone. > > > > To conclude, having `kie-tools` building a "template image" to verify > > that it will not break `kogito-serverless-operator` downstream, and > > having `kogito-serverless-operator` building parts of `kie-tools` to > > incorporate the latest changes, IMHO, is: > > 1. Duplication of building logic. IMHO, we can't have individual repos > > creating its own up- and downstream CI systems. > > 2. Conflicting with the timestamped SNAPSHOTs strategy for integrating > > repos, where boundaries are clearly defined. > > 3. Wasteful use of resources (same that is happening with > > `kogito-images` building `kogito-apps` right now). > > > > I'm sorry for bringing all this to this seemingly simple proposal, but > > I'm afraid taking steps without knowing where we're going isn't the > > way to move forward. > > > > Please let me know if I misunderstood something. > > > > On Wed, May 8, 2024 at 3:15 PM ricardo zanini fernandes > > <[email protected]> wrote: > > > > > > Hi! > > > > > > Sorry for the late reply, I was on PTO. > > > > > > Replies inline. > > > > > > > The `kie-tools` CI is not prepared to build > > > `kogito-serverless-operator`, so the way `kogito-serverless-operator` > > > references the SonataFlow Quarkus Dev UI will be important to > > > establish the boundaries between both repos. To further develop the > > > SonataFlow Quarkus Dev UI and have its changes reflect on the > > > `kogito-swf-devmode` image, we need to come up with a strategy that is > > > both safe, consistent, and enforces correctness. There's also the fact > > > that currently `kie-tools` depends on timestamped SNAPSHOTs of > > > Kogito/Drools, while `kogito-serverless-operator` uses 999-SNAPSHOT, > > > if I'm not mistaken. Can you elaborate a little bit more on how you > > > see this reference being done? Please consider cross-repo PRs for big > > > migrations like the Quarkus 3.8.4 that is currently happening. > > > > > > This will make us discuss yet again the need of snapshots for the UI > > > components used by the images. The images must be the end of the build > > > pipeline, where we aggregate every component we ship in a single > instance > > > to release. Yet, they are a core part of the cloud platform, hence > part of > > > the Operator BDD, testing, integration, and delivering. Intermediate > repos > > > can't depend on them unless they are also responsible for maintenance > and > > > release like what we're doing with the consoles. > > > > > > Also, the images can have a respin anytime we need a new component > bump. We > > > do this all the time to fix CVEs or bug fixes in a specific component > that > > > is part of the image. > > > > > > The devui can be developed and maintained within the kie-tools repo and > > > have tests that verify that component. The integration will be made on > the > > > Images/Operator side once we grab a latest snapshot. Alternatively, we > can > > > do: > > > 1. The tools CI can fetch a template image, inject the component, and > run > > > the required tests > > > 2. The image/operator CI can fetch and build the UI components and run > the > > > integration tests > > > > > > > Also, regarding point "4. Review the PR GHA checks to run CLI tests > > > once there's a change in the cmd module" of the proposed EPIC, I think > > > we might run into problems, since the `cmd` module also depends on the > > > `api` and `workflowproj` modules of `kogito-serverless-operator.` I'm > > > afraid changes made to these two modules would also need to trigger a > > > build of the `cmd` module, and they can potentially break it. > > > > > > Sorry to not get into details there, but the new cmd module will be > part of > > > the Operator's workspace, any changes in any cross dependencies there > will > > > trigger the CI checks, as we are doing today in the builder and > > > workflowproj modules. It will be way easier to maintain the CLI. > > > > > > > These considerations alone, IMHO, expose one of the biggest > challenges > > > we have in our community right now, which is that the definition and > > > implementation of the dependency graph between repos/modules/packages > > > is currently spread across many different "build systems", like the > > > new proposed GHA jobs exclusive to the `kogito-serverless-operator` > > > repo, the Build Chain system we have for the Drools/Kogito repos, the > > > `kie-tools` CI, and the many Jenkins jobs we have on Apache's Jenkins. > > > There's also the fact that we have `kogito-images` selectively > > > building parts of `kogito-apps` during its own build to include them > > > in the images it produces. > > > > > > I agree. We would need to sit together and solve this as a team, > having a > > > nice integration across every repo. The kogito-images building parts > of the > > > apps is something we currently working on, IIRC. The apps will deploy a > > > snapshot as part of the nightlies and the images will be using it. > > > > > > > I just think it is important to highlight that this proposal would > > > only address a LOCAL problem related exclusively to the SonataFlow > > > section of the Apache KIE community, while, at the same time, not > > > contributing to reducing the segmentation of the Apache KIE community > > > as a whole. > > > > > > Yes, the proposal is to starting addressing the local Apache KIE > SonataFlow > > > Cloud platform first. But I agree that we need to refactor our CI as a > > > whole, which is something we should start in a new thread. > > > > > > Cheers! > > > > > > On Mon, Apr 22, 2024 at 10:02 PM Tiago Bento <[email protected]> > wrote: > > > > > > > Zanini and Alex, > > > > > > > > The task we agreed on for after releasing Apache KIE 10 is > > > > https://github.com/apache/incubator-kie-issues/issues/1040. It only > > > > describes deleting the temporary copies we'll have on KIE Tools and > > > > reverting things back to where they were, using the fixed 10.0.0 > > > > version. > > > > > > > > Moving `kn-plugin-workflow` and the `kogito-swf-{devmode,builder}` > > > > images to `kogito-serverless-operator` would be a new move, which I > > > > understand is the scope of this proposal thread. > > > > > > > > Doing so, however, will make `kogito-serverless-operator` depend on > > > > `kie-tools`, since the SonataFlow Quarkus Dev UI is hosted there now, > > > > and it is a dependency of the `kogito-swf-devmode` image. > > > > > > > > I'm saying this because I think we need to further discuss the > > > > consequences of this change... > > > > > > > > The `kie-tools` CI is not prepared to build > > > > `kogito-serverless-operator`, so the way `kogito-serverless-operator` > > > > references the SonataFlow Quarkus Dev UI will be important to > > > > establish the boundaries between both repos. To further develop the > > > > SonataFlow Quarkus Dev UI and have its changes reflect on the > > > > `kogito-swf-devmode` image, we need to come up with a strategy that > is > > > > both safe, consistent, and enforces correctness. There's also the > fact > > > > that currently `kie-tools` depends on timestamped SNAPSHOTs of > > > > Kogito/Drools, while `kogito-serverless-operator` uses 999-SNAPSHOT, > > > > if I'm not mistaken. Can you elaborate a little bit more on how you > > > > see this reference being done? Please consider cross-repo PRs for big > > > > migrations like the Quarkus 3.8.4 that is currently happening. > > > > > > > > Also, regarding point "4. Review the PR GHA checks to run CLI tests > > > > once there's a change in the cmd module" of the proposed EPIC, I > think > > > > we might run into problems, since the `cmd` module also depends on > the > > > > `api` and `workflowproj` modules of `kogito-serverless-operator.` I'm > > > > afraid changes made to these two modules would also need to trigger a > > > > build of the `cmd` module, and they can potentially break it. > > > > > > > > These considerations alone, IMHO, expose one of the biggest > challenges > > > > we have in our community right now, which is that the definition and > > > > implementation of the dependency graph between repos/modules/packages > > > > is currently spread across many different "build systems", like the > > > > new proposed GHA jobs exclusive to the `kogito-serverless-operator` > > > > repo, the Build Chain system we have for the Drools/Kogito repos, the > > > > `kie-tools` CI, and the many Jenkins jobs we have on Apache's > Jenkins. > > > > There's also the fact that we have `kogito-images` selectively > > > > building parts of `kogito-apps` during its own build to include them > > > > in the images it produces. > > > > > > > > With all that said, I'm not opposed to moving the > `kn-workflow-plugin` > > > > package from `kie-tools` to the `cmd` module of > > > > `kogito-serverless-operator`. In fact, like I said in the past, I > > > > think it makes a lot of sense that they're part of the same > > > > compilation unit, as they're very closely related, and should > > > > therefore be in sync at all times. > > > > > > > > I just think it is important to highlight that this proposal would > > > > only address a LOCAL problem related exclusively to the SonataFlow > > > > section of the Apache KIE community, while, at the same time, not > > > > contributing to reducing the segmentation of the Apache KIE community > > > > as a whole. > > > > > > > > Regards, > > > > > > > > Tiago Bento > > > > > > > > On Fri, Apr 19, 2024 at 4:08 PM ricardo zanini fernandes > > > > <[email protected]> wrote: > > > > > > > > > > Alex, > > > > > > > > > > Yes, in the proposal we just barely outlined. I create the EPIC to > have > > > > > more details and start working on them as soon as we agree. > > > > > > > > > > On Fri, Apr 19, 2024 at 4:24 PM Alex Porcelli <[email protected]> > wrote: > > > > > > > > > > > Thank you for outlining the tasks post the 10.x release. It's > > > > > > important to note that these are already included in the amended > > > > > > proposal [1], specifically in steps 9 and 10, which the > community has > > > > > > voted on. If there are concerns about the execution of these > steps, > > > > > > I'd like to reassure you that they will proceed as planned, in > > > > > > compliance with Apache guidelines. > > > > > > > > > > > > Looking ahead, after the release of version 10, we already > agreed that > > > > > > we'll need to have a thorough discussion regarding the codebase > > > > > > structure. This will allow us to refine our understanding of the > > > > > > sub-brands, their interrelationships, and their strategic > positioning. > > > > > > I agree that this is crucial for our next steps and look forward > to > > > > > > our collaborative efforts in shaping this. > > > > > > > > > > > > [1] - > https://lists.apache.org/thread/pw327lkpmy9gxklq4t5zbwzxjo2y3c0w > > > > > > > > > > > > On Fri, Apr 19, 2024 at 2:50 PM ricardo zanini fernandes > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > I've outlined the tasks we need once we release 10.x from > kie-tools: > > > > > > > https://github.com/apache/incubator-kie-issues/issues/1102 > > > > > > > > > > > > > > Once we release, we can detail this planning and start working > on it > > > > to > > > > > > > have a streamlined process for the next release. > > > > > > > > > > > > > > Please let me know if it makes sense to you. We can break down > and > > > > detail > > > > > > > the tasks once we agree on this initial plan. > > > > > > > > > > > > > > Cheers! > > > > > > > -- > > > > > > > Ricardo Zanini Fernandes > > > > > > > Vida longa e próspera. > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
