Hi Tiago, I would argue that, being all the repos part of the same release, mutable snapshot is perfectly acceptable. As it currently happens between Runtimes repo and Drools repo (which have the Kie api) and Apps repo and Runtimes repo. I, as I believe Enrique (at least if what I understood from his e-mail) , do not understand why tooling should be different in that regard.
On Fri, May 10, 2024 at 1:26 AM Tiago Bento <tiagobe...@apache.org> wrote: > Zanini and Enrique, > > Not sure if I'm making it sound differently, but everything I'm saying > is also aiming towards a healthy CI. Not only for SonataFlow or Tools, > though. For all of Apache KIE. > > My main point is that mutable Maven SNAPSHOTs are harming us, giving a > false impression of synchronization between the repos. It's an "easy > way out" we've been historically abusing, IMHO. > > One of the best advantages of having multiple repos is that they can > be individually developed, tested, and managed. However, by using > Maven SNAPSHOTs as the synchronization mechanism, we can't guarantee > that, since repos become broken on any given day. More than that, it's > impossible to go "back in time" on our development branches, since the > SNAPSHOT version will always point to latest. > > Please read this section from the guidelines of the SciJava project at > https://github.com/scijava/pom-scijava-base/blob/main/README.md. > > "Reproducible builds. This rule means no SNAPSHOT dependencies, no > SNAPSHOT parents, and no SNAPSHOT plugin versions. A snapshot version > is not immutable, which means that code which depends on a snapshot > may build today, but not build tomorrow, if the snapshot is later > changed. The best way to avoid this conundrum is to never depend on > SNAPSHOT versions. Snapshot are best used for testing only; they can > be used transiently, but their use should never make it onto the main > integration branch (e.g., main or master) of a project. See also Using > snapshot couplings during development at > > https://imagej.net/develop/architecture#using-snapshot-couplings-during-development > ." > > This summarizes very well the problem I'm trying to convey here. Of > course, their words are much more clear than all the explanations I > tried giving so far. > > > > > On Thu, May 9, 2024 at 9:43 AM ricardo zanini fernandes > <ricardozan...@gmail.com> wrote: > > > > Tiago, > > > > Again, this is a major problem in our current structure that we should > use > > another thread to discuss. For my proposal is to at least have a health > > cloud build platform. We can completely remove UI from our images in the > > SonataFlow space and keep the consoles in kie-tools. Once we solve the > deep > > structural problem we have, we can evaluate it better. What we can't do > is > > be behind and not release. > > > > The template image is pretty useful if you really want to test the DevUI > in > > the image. You can easily pull the container file and build the image > > during your integration tests and run the tests you need. We can do the > > same in the image. I don't see a waste of resources, but as a temporary > > alternative until we figure out the deeper problem. > > > > At the moment, everything "tools" related is in a single repo and the > other > > components are scattered in many other repos. We have to decide if we > want > > mono repos (infra-oriented) or context repos (feature-oriented). We can't > > have both otherwise the mono repos will start to drag others to it, as we > > are seeing happening. > > > > So IMO, we need first to sit down together and have a new repo structure > > design, perhaps based on components. We need to evaluate. > > > > For the time being, we need to move forward at least and I can live with > > integration tests at the end of the line using final versions/built > > versions of the DevUI if needed for SonataFlow. > > > > > > > > On Thu, May 9, 2024 at 10:32 AM ricardo zanini fernandes < > > ricardozan...@gmail.com> wrote: > > > > > Hi Enrique! > > > > > > The images are using SNAPSHOTS from apps/runtimes currently. There was > a > > > decision in the past to build apps in the images from the > main/versioned > > > branch. We are changing this to use SNAPSHOT/versioned artifacts > published > > > in Maven. About kie-tools, I don't know either. > > > > > > We can use immutable snapshots (weekly, nightlies) instead. I'm not > > > against it. But we do need those for integration tests or at least a > robust > > > CI platform to do integration without snapshots. > > > > > > Cheers! > > > > > > On Thu, May 9, 2024 at 5:07 AM Enrique Gonzalez Martinez < > > > egonza...@apache.org> wrote: > > > > > >> Hi Tiago, > > >> > > >> I think the discussion is related to the graph dependency for now. So > > >> I would keep that in mind. Anyway I am kinda surprised about some > > >> concepts like (mutable snapshots and using snapshot published > > >> timestamps like they are final). > > >> > > >> Now I think I understand where the problem lies now; so basically if > > >> we had the proper graph dependency project built based on proper > > >> snapshots everything would build just ok. So my question is pretty > > >> simple, why is it not possible for kie tools or kogito images to use > > >> the maven snapshot concept and point out to the last snapshot build > > >> (either local or remote) like other maven projects work. Is there any > > >> limitation there ? > > >> > > >> El mié, 8 may 2024 a las 22:30, Tiago Bento (<tiagobe...@apache.org>) > > >> escribió: > > >> > > > >> > Hi Zanini! Welcome back, hope you had a great time during your PTO. > > >> > > > >> > If I understand correctly, the way you see the integration between > > >> > repos relies heavily on timestamped SNAPSHOTs. This way: > > >> > 1. `kogito-images` would stop building `kogito-apps`, and would rely > > >> > on timestamped SNAPSHOTs being published from > > >> > `drools/optaplanner/kogito-runtimes/kogito-apps`. > > >> > 2. `kie-tools` would start publishing timestamped SNAPSHOTs for its > UI > > >> > components, so they would be consumed downstream by > > >> > `kogito-serverless-operator`. > > >> > > > >> > If that's what you mean, let me elaborate on some of the > consequences > > >> > of this integration model, where we stop using the mutable > > >> > 999-SNAPSHOT and start relying more on timestamped SNAPSHOTs for > > >> > cross-repo synchronization: > > >> > a. Cross-repo PRs would not be possible anymore, depending on which > > >> > repos are involved in the ensemble. This would create boundaries > > >> > between certain groups of our repos, namely: > > >> > drools/optaplanner/kogito-runtimes/kogito-apps --> kogito-images --> > > >> > kie-tools, --> kogito-serverless-operator. So 4 "independent > > >> > clusters". > > >> > b. kogito-images, kie-tools, and kogito-serverless-operator, using > > >> > timestamped SNAPSHOTs between themselves, would make the most > upstream > > >> > of the three (in this case, `kogito-images`), to be the one defining > > >> > what timestamped SNAPSHOT version we should use for its upstream > > >> > dependencies (drools/optaplanner/kogito-runtimes/kogito-apps). In > more > > >> > details: If `kogito-images` depends on > > >> > `drools/optaplanner/kogito-runtimes/kogito-apps` using version > > >> > 999-20240501, that's the version `kie-tools` and > > >> > `kogito-serverless-operator` need to use as well, as the > > >> > drools/optaplanner/kogito-runtimes/kogito-apps would come > transitively > > >> > from the `kogito-images` dependency these two repos would have. > This, > > >> > IMHO, causes a lot of confusion, since the dates on the timestamped > > >> > SNAPSHOT versions have implicit transitive dependencies to other > > >> > upstream projects that also have their own timestamped SNAPSHOTs > > >> > published. I.e., 999-20240501 of `kie-tools` won't be aligned with > > >> > 999-2024051 of `drools/optaplanner/kogito-runtimes/kogito-apps`. > > >> > c. Updating dependencies across the board would be much more > > >> > difficult, as we would need to rely on a chain reaction starting > from > > >> > the `drools/optaplanner/kogito-runtimes/kogito-apps` cluster all the > > >> > way down to `kogito-serverless-operator`, releasing timestamped > > >> > SNAPSHOTs and sending PRs for each "cluster" along the way. > > >> > > > >> > I'm not sure this is the way we want to move forward, to be honest, > as > > >> > these consequences need to be thoroughly examined by people in the > > >> > community, so that all the side-effects of this choice are clear to > > >> > everyone. > > >> > > > >> > To conclude, having `kie-tools` building a "template image" to > verify > > >> > that it will not break `kogito-serverless-operator` downstream, and > > >> > having `kogito-serverless-operator` building parts of `kie-tools` to > > >> > incorporate the latest changes, IMHO, is: > > >> > 1. Duplication of building logic. IMHO, we can't have individual > repos > > >> > creating its own up- and downstream CI systems. > > >> > 2. Conflicting with the timestamped SNAPSHOTs strategy for > integrating > > >> > repos, where boundaries are clearly defined. > > >> > 3. Wasteful use of resources (same that is happening with > > >> > `kogito-images` building `kogito-apps` right now). > > >> > > > >> > I'm sorry for bringing all this to this seemingly simple proposal, > but > > >> > I'm afraid taking steps without knowing where we're going isn't the > > >> > way to move forward. > > >> > > > >> > Please let me know if I misunderstood something. > > >> > > > >> > On Wed, May 8, 2024 at 3:15 PM ricardo zanini fernandes > > >> > <ricardozan...@gmail.com> wrote: > > >> > > > > >> > > Hi! > > >> > > > > >> > > Sorry for the late reply, I was on PTO. > > >> > > > > >> > > Replies inline. > > >> > > > > >> > > > The `kie-tools` CI is not prepared to build > > >> > > `kogito-serverless-operator`, so the way > `kogito-serverless-operator` > > >> > > references the SonataFlow Quarkus Dev UI will be important to > > >> > > establish the boundaries between both repos. To further develop > the > > >> > > SonataFlow Quarkus Dev UI and have its changes reflect on the > > >> > > `kogito-swf-devmode` image, we need to come up with a strategy > that is > > >> > > both safe, consistent, and enforces correctness. There's also the > fact > > >> > > that currently `kie-tools` depends on timestamped SNAPSHOTs of > > >> > > Kogito/Drools, while `kogito-serverless-operator` uses > 999-SNAPSHOT, > > >> > > if I'm not mistaken. Can you elaborate a little bit more on how > you > > >> > > see this reference being done? Please consider cross-repo PRs for > big > > >> > > migrations like the Quarkus 3.8.4 that is currently happening. > > >> > > > > >> > > This will make us discuss yet again the need of snapshots for the > UI > > >> > > components used by the images. The images must be the end of the > build > > >> > > pipeline, where we aggregate every component we ship in a single > > >> instance > > >> > > to release. Yet, they are a core part of the cloud platform, hence > > >> part of > > >> > > the Operator BDD, testing, integration, and delivering. > Intermediate > > >> repos > > >> > > can't depend on them unless they are also responsible for > maintenance > > >> and > > >> > > release like what we're doing with the consoles. > > >> > > > > >> > > Also, the images can have a respin anytime we need a new component > > >> bump. We > > >> > > do this all the time to fix CVEs or bug fixes in a specific > component > > >> that > > >> > > is part of the image. > > >> > > > > >> > > The devui can be developed and maintained within the kie-tools > repo > > >> and > > >> > > have tests that verify that component. The integration will be > made > > >> on the > > >> > > Images/Operator side once we grab a latest snapshot. > Alternatively, > > >> we can > > >> > > do: > > >> > > 1. The tools CI can fetch a template image, inject the component, > and > > >> run > > >> > > the required tests > > >> > > 2. The image/operator CI can fetch and build the UI components and > > >> run the > > >> > > integration tests > > >> > > > > >> > > > Also, regarding point "4. Review the PR GHA checks to run CLI > tests > > >> > > once there's a change in the cmd module" of the proposed EPIC, I > think > > >> > > we might run into problems, since the `cmd` module also depends > on the > > >> > > `api` and `workflowproj` modules of `kogito-serverless-operator.` > I'm > > >> > > afraid changes made to these two modules would also need to > trigger a > > >> > > build of the `cmd` module, and they can potentially break it. > > >> > > > > >> > > Sorry to not get into details there, but the new cmd module will > be > > >> part of > > >> > > the Operator's workspace, any changes in any cross dependencies > there > > >> will > > >> > > trigger the CI checks, as we are doing today in the builder and > > >> > > workflowproj modules. It will be way easier to maintain the CLI. > > >> > > > > >> > > > These considerations alone, IMHO, expose one of the biggest > > >> challenges > > >> > > we have in our community right now, which is that the definition > and > > >> > > implementation of the dependency graph between > repos/modules/packages > > >> > > is currently spread across many different "build systems", like > the > > >> > > new proposed GHA jobs exclusive to the > `kogito-serverless-operator` > > >> > > repo, the Build Chain system we have for the Drools/Kogito repos, > the > > >> > > `kie-tools` CI, and the many Jenkins jobs we have on Apache's > Jenkins. > > >> > > There's also the fact that we have `kogito-images` selectively > > >> > > building parts of `kogito-apps` during its own build to include > them > > >> > > in the images it produces. > > >> > > > > >> > > I agree. We would need to sit together and solve this as a team, > > >> having a > > >> > > nice integration across every repo. The kogito-images building > parts > > >> of the > > >> > > apps is something we currently working on, IIRC. The apps will > deploy > > >> a > > >> > > snapshot as part of the nightlies and the images will be using it. > > >> > > > > >> > > > I just think it is important to highlight that this proposal > would > > >> > > only address a LOCAL problem related exclusively to the SonataFlow > > >> > > section of the Apache KIE community, while, at the same time, not > > >> > > contributing to reducing the segmentation of the Apache KIE > community > > >> > > as a whole. > > >> > > > > >> > > Yes, the proposal is to starting addressing the local Apache KIE > > >> SonataFlow > > >> > > Cloud platform first. But I agree that we need to refactor our CI > as a > > >> > > whole, which is something we should start in a new thread. > > >> > > > > >> > > Cheers! > > >> > > > > >> > > On Mon, Apr 22, 2024 at 10:02 PM Tiago Bento < > tiagobe...@apache.org> > > >> wrote: > > >> > > > > >> > > > Zanini and Alex, > > >> > > > > > >> > > > The task we agreed on for after releasing Apache KIE 10 is > > >> > > > https://github.com/apache/incubator-kie-issues/issues/1040. It > only > > >> > > > describes deleting the temporary copies we'll have on KIE Tools > and > > >> > > > reverting things back to where they were, using the fixed 10.0.0 > > >> > > > version. > > >> > > > > > >> > > > Moving `kn-plugin-workflow` and the > `kogito-swf-{devmode,builder}` > > >> > > > images to `kogito-serverless-operator` would be a new move, > which I > > >> > > > understand is the scope of this proposal thread. > > >> > > > > > >> > > > Doing so, however, will make `kogito-serverless-operator` > depend on > > >> > > > `kie-tools`, since the SonataFlow Quarkus Dev UI is hosted there > > >> now, > > >> > > > and it is a dependency of the `kogito-swf-devmode` image. > > >> > > > > > >> > > > I'm saying this because I think we need to further discuss the > > >> > > > consequences of this change... > > >> > > > > > >> > > > The `kie-tools` CI is not prepared to build > > >> > > > `kogito-serverless-operator`, so the way > > >> `kogito-serverless-operator` > > >> > > > references the SonataFlow Quarkus Dev UI will be important to > > >> > > > establish the boundaries between both repos. To further develop > the > > >> > > > SonataFlow Quarkus Dev UI and have its changes reflect on the > > >> > > > `kogito-swf-devmode` image, we need to come up with a strategy > that > > >> is > > >> > > > both safe, consistent, and enforces correctness. There's also > the > > >> fact > > >> > > > that currently `kie-tools` depends on timestamped SNAPSHOTs of > > >> > > > Kogito/Drools, while `kogito-serverless-operator` uses > 999-SNAPSHOT, > > >> > > > if I'm not mistaken. Can you elaborate a little bit more on how > you > > >> > > > see this reference being done? Please consider cross-repo PRs > for > > >> big > > >> > > > migrations like the Quarkus 3.8.4 that is currently happening. > > >> > > > > > >> > > > Also, regarding point "4. Review the PR GHA checks to run CLI > tests > > >> > > > once there's a change in the cmd module" of the proposed EPIC, I > > >> think > > >> > > > we might run into problems, since the `cmd` module also depends > on > > >> the > > >> > > > `api` and `workflowproj` modules of > `kogito-serverless-operator.` > > >> I'm > > >> > > > afraid changes made to these two modules would also need to > trigger > > >> a > > >> > > > build of the `cmd` module, and they can potentially break it. > > >> > > > > > >> > > > These considerations alone, IMHO, expose one of the biggest > > >> challenges > > >> > > > we have in our community right now, which is that the > definition and > > >> > > > implementation of the dependency graph between > > >> repos/modules/packages > > >> > > > is currently spread across many different "build systems", like > the > > >> > > > new proposed GHA jobs exclusive to the > `kogito-serverless-operator` > > >> > > > repo, the Build Chain system we have for the Drools/Kogito > repos, > > >> the > > >> > > > `kie-tools` CI, and the many Jenkins jobs we have on Apache's > > >> Jenkins. > > >> > > > There's also the fact that we have `kogito-images` selectively > > >> > > > building parts of `kogito-apps` during its own build to include > them > > >> > > > in the images it produces. > > >> > > > > > >> > > > With all that said, I'm not opposed to moving the > > >> `kn-workflow-plugin` > > >> > > > package from `kie-tools` to the `cmd` module of > > >> > > > `kogito-serverless-operator`. In fact, like I said in the past, > I > > >> > > > think it makes a lot of sense that they're part of the same > > >> > > > compilation unit, as they're very closely related, and should > > >> > > > therefore be in sync at all times. > > >> > > > > > >> > > > I just think it is important to highlight that this proposal > would > > >> > > > only address a LOCAL problem related exclusively to the > SonataFlow > > >> > > > section of the Apache KIE community, while, at the same time, > not > > >> > > > contributing to reducing the segmentation of the Apache KIE > > >> community > > >> > > > as a whole. > > >> > > > > > >> > > > Regards, > > >> > > > > > >> > > > Tiago Bento > > >> > > > > > >> > > > On Fri, Apr 19, 2024 at 4:08 PM ricardo zanini fernandes > > >> > > > <ricardozan...@gmail.com> wrote: > > >> > > > > > > >> > > > > Alex, > > >> > > > > > > >> > > > > Yes, in the proposal we just barely outlined. I create the > EPIC > > >> to have > > >> > > > > more details and start working on them as soon as we agree. > > >> > > > > > > >> > > > > On Fri, Apr 19, 2024 at 4:24 PM Alex Porcelli < > a...@porcelli.me> > > >> wrote: > > >> > > > > > > >> > > > > > Thank you for outlining the tasks post the 10.x release. > It's > > >> > > > > > important to note that these are already included in the > amended > > >> > > > > > proposal [1], specifically in steps 9 and 10, which the > > >> community has > > >> > > > > > voted on. If there are concerns about the execution of these > > >> steps, > > >> > > > > > I'd like to reassure you that they will proceed as planned, > in > > >> > > > > > compliance with Apache guidelines. > > >> > > > > > > > >> > > > > > Looking ahead, after the release of version 10, we already > > >> agreed that > > >> > > > > > we'll need to have a thorough discussion regarding the > codebase > > >> > > > > > structure. This will allow us to refine our understanding > of the > > >> > > > > > sub-brands, their interrelationships, and their strategic > > >> positioning. > > >> > > > > > I agree that this is crucial for our next steps and look > > >> forward to > > >> > > > > > our collaborative efforts in shaping this. > > >> > > > > > > > >> > > > > > [1] - > > >> https://lists.apache.org/thread/pw327lkpmy9gxklq4t5zbwzxjo2y3c0w > > >> > > > > > > > >> > > > > > On Fri, Apr 19, 2024 at 2:50 PM ricardo zanini fernandes > > >> > > > > > <ricardozan...@gmail.com> wrote: > > >> > > > > > > > > >> > > > > > > Folks, > > >> > > > > > > > > >> > > > > > > I've outlined the tasks we need once we release 10.x from > > >> kie-tools: > > >> > > > > > > > https://github.com/apache/incubator-kie-issues/issues/1102 > > >> > > > > > > > > >> > > > > > > Once we release, we can detail this planning and start > > >> working on it > > >> > > > to > > >> > > > > > > have a streamlined process for the next release. > > >> > > > > > > > > >> > > > > > > Please let me know if it makes sense to you. We can break > > >> down and > > >> > > > detail > > >> > > > > > > the tasks once we agree on this initial plan. > > >> > > > > > > > > >> > > > > > > Cheers! > > >> > > > > > > -- > > >> > > > > > > Ricardo Zanini Fernandes > > >> > > > > > > Vida longa e próspera. > > >> > > > > > > > >> > > > > > > > >> --------------------------------------------------------------------- > > >> > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > >> > > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> --------------------------------------------------------------------- > > >> > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > >> > > > For additional commands, e-mail: dev-h...@kie.apache.org > > >> > > > > > >> > > > > > >> > > > >> > > --------------------------------------------------------------------- > > >> > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > >> > For additional commands, e-mail: dev-h...@kie.apache.org > > >> > > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > >> For additional commands, e-mail: dev-h...@kie.apache.org > > >> > > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > For additional commands, e-mail: dev-h...@kie.apache.org > >
