Hi Tiago El vie, 10 may 2024, 17:22, Tiago Bento <[email protected]> escribió:
> Enrique, > > Let me give you an example. > > Yesterday, a module from `kogito-apps` had a problem with a data > source configuration for `data-index-inmemory`. This module is used on > `kie-tools`, and is part of some integration tests that we run. > Despite that module being broken on `kogito-apps`, the `kie-tools` > build and tests continued green, and no contributions stopped because > of it. This is only possible because `kie-tools` depends on > `kogito-apps` through a fixed, immutable, timestamped SNAPSHOT > (currently at 999-20240417-SNAPSHOT), which is essentially the same as > a fixed version. If `kie-tools` was using `999-SNAPSHOT`, `kie-tools` > would've been broken for who knows how long until that problem was > fixed. > > The fact that we're depending on a fixed, immutable, timestamped > SNAPSHOT, allows us to wait for a fix to be available so we > incorporate the latest changes. That's what this PR is doing --> > https://github.com/apache/incubator-kie-tools/pull/2272. Right now, `kie-tools` is the only repo using that strategy, because > `kie-tools` is the last of the train. If we were to proceed with > Zanini's proposal on this email, `kogito-serverless-operator` would be > the last one, and it would suffer the same problems. > > Breaking downstream repos is NOT acceptable. Of course no one does it intentionally, but when something breaks > somewhere, we can't put all the downstream projects to a halt. > > If you're working on `kogito-runtimes` and never compiling `drools`, it's because people contributing to `drools` are doing a fantastic job > in keeping things stable, or maybe because there aren't as many big > things happening in `drools` right now. I wouldn't know... However, > the moment things start being renamed, APIs start to change and big > refactors take place, I can guarantee you'll find yourself building > `drools` everyday before working on `kogito-runtimes`. We can't discourage changes because it might break downstream. I need > to put mechanisms in place that will allow us to do changes without > fear of stopping the contributions. More than that, we need to be able > to continue operating when an unintentional mistake happens. Hi Tiago, Now it is crystal clear that the problem is not really CI problem but the disruption in tooling workflow because there are no proper checks. This wont change no matter what CI or build we use. It is really that simple. Because it is check problem, not a CI problem. Regarding the explanation this is not something special in tooling. We all deal with breaking changes because a pr was merged and you were working under certain premises. Tooling just got it worse because as you said it is the last one of the train. I can understand the pain in that regard and we all acknowledge that. No discussion in there. I understand that using a completely different tech requires different workflow and different approaches to work through the day but tooling cannot expect other teams to be aware of what they are using or doing. The project is far to big to keep taps on everything is happening. Drools team does not impact other repos because they run all the checks so they are more less safe they wont impact jbpm, runtimes or apps therefore It is safe to merge if they dont break any unit test. It is not silver bullet though. So yes we jbpm,runtimes and apps are responsible for checking that our integration with drools works the way it needs to work. Because drools team does not know in deep detail how we are using them to a deep level. So yes, your integrations with other technologies needs to be checked. > On Fri, May 10, 2024 at 9:40 AM Enrique Gonzalez Martinez > <[email protected]> wrote: > > > > Hi Tiago, > > > > You need to perform the checks no matter what the structure is, > > otherwise you will have problems breaking something when you push > > code. You cannot expect backend to not break anything if we don't get > > the checks from tooling. > > Breaking/disrupting some other repo is not a special case. We all have > > that problem. It is true that unit testing and integration tests are > > not silver bullets. > > Regarding snapshots is the way it works with maven and it is a proper > > lifecycle (you need some sort of artifact in the middle ground). Again > > snapshots are not final, snapshots are just successfully builds > > Git history and snapshots are totally different things ( I guess you > > are talking about you cannot match a snapshot with the commit hash > > that was build, it is possible but not sure what you want that) but it > > is not something relevant. > > I can work in kogito-runtimes and never compile drools because it is > > not required as the nightly build works as cached build for core > > engineers. So this is not really a problem in 95 % of the cases unless > > you have a multirepo PR. > > > > What it is important to me is related to somethings were said here. > > > > 1. kie tools can consume snapshots without problem therefore the build > > pipeline is fixed or can be fixed to perform proper build. > > 2. some tools cannot produce snapshots but that is a limitation > > 3. seems that the painful point is work disruption when us (core > > engineers) perform changes that can break tooling > > > > is that correct ? > > > > El vie, 10 may 2024 a las 13:32, Tiago Bento > > (<[email protected]>) escribió: > > > > > > Doubling down on SNAPSHOTs will make everyone’s life worse. CI is > going to > > > take forever to run for anyone doing changes, and everyone will have to > > > build the upstream stack every morning, as the published SNAPSHOTs can > be > > > stale by the time each of us wakes up to work. This is going back to > > > Business Central days, and I’m pretty sure we don’t want to repeat > > > ourselves in that regard. > > > > > > In short, SNAPSHOTs are unreliable for intra-day work, and useless for > > > pretty much anything else that we’re currently doing. > > > > > > If you’re working on kogito-apps and you don’t build drools and > > > kogito-runtimes, you’ll most likely have a broken build. Especially > during > > > times when there’s a lot going on, like the last couple of months. > > > > > > Please also don’t ignore the fact that SNAPSHOTs make it impossible to > > > correctly travel our Git history. > > > > > > On Fri, May 10, 2024 at 7:17 AM Alex Porcelli <[email protected]> > wrote: > > > > > > > Enrique, > > > > > > > > If CI is completely redone and we have unified checks on every PR > for every > > > > change covering all repos, yes… tools could use snapshot for > consumption. > > > > > > > > > > > > However there isn’t such a thing as snapshot for other technologies > like > > > > npm packages. Anything that has any UI dependency has to be in the > same > > > > repository, as internal consumption is possible… just not possible > > > > externally without a proper SEMVER release (snapshot is not SEMVER). > > > > > > > > > > > > On Fri, May 10, 2024 at 7:05 AM Enrique Gonzalez Martinez < > > > > [email protected]> wrote: > > > > > > > > > Hi Gabriele > > > > > > > > > > El vie, 10 may 2024, 12:57, Gabriele Cardosi < > [email protected] > > > > > > > > > > escribió: > > > > > > > > > > > Hi Enrique, > > > > > > I'm ignoring anything else, but regarding > > > > > > > > > > > > "The other topic you are talking is the build. The question was > > > > > summarized > > > > > > by Francisco.if we build the repos in order, does kie tool able > to > > > > build > > > > > > properly with the snapshots built in the pipeline or does need > > > > additional > > > > > > steps ?" > > > > > > > > > > > > I think the core of the problem is the following: > > > > > > when we backender made a modification (e.g. in drools repo) we > are > > > > > somehow > > > > > > responsible to fix eventual break downstream (e.g. > kogito-runtimes, > > > > > > kogito-examples, etc), but we completely ignores the tooling, > and we > > > > > won't > > > > > > probably be able to fix it. So, the tooling guys must rely on > some > > > > > "stable" > > > > > > version, otherwise we could break their work on a daily basis. > > > > > > This is more or less the situation I found when I joined: the > drools > > > > > built > > > > > > was separated from the drools-wb one, sometimes modification on > drools > > > > > repo > > > > > > broke drools-wb, and then they decided to pipeline the twos, to > have > > > > > > immediate recognition, and fix, of that (at that time, our > drools-wb > > > > job > > > > > > was to build drools every morning and then start working on > drools-wb: > > > > if > > > > > > something get broken in drools-wb, then we pinged drool devs to > > > > > understand > > > > > > reason and eventually make fix). > > > > > > The solutions I see for our situation are: > > > > > > > > > > > > 1. keep tooling depend on some stable release, preserving > > > > independence > > > > > > from backender changes > > > > > > 2. enforce/improve cooperation between backender and tooling > guys, > > > > so > > > > > > that as soon as some breaking change is made by the former, > the > > > > latter > > > > > > can > > > > > > immediately fix > > > > > > > > > > > > > > > > > > > > > This is not a special scenario for anyone. This happens as well for > > > > others > > > > > repo. If you break and pr does not pass u redo your PR. > > > > > I don't see anything special there. So if the only thing for > tooling to > > > > set > > > > > a specific version of a snapshot it means we have a problem into > our unit > > > > > testing. It is impossible for backenders to fix something we are > not > > > > aware > > > > > it is broken because the pr checks are not in there. At some point > you > > > > need > > > > > you rely on CI. We all have to live with that problem. But at > least we > > > > > should have the pr checks in place. That should mitigate the > problem > > > > about > > > > > breaking stuff in other places. > > > > > > > > > > > > > > > Still this not answers my question regarding the building. It just > > > > explains > > > > > why the build might break something in tooling because our pr > cheks are > > > > not > > > > > in place. > > > > > > > > > > > > > > > > > > > > > Side notes: > > > > > > > > > > > > 1. About point 1, I do not think it breaks the "single > product" > > > > rule, > > > > > > because at release time everything should be aligned > > > > > > 2. About point 2, I'm not sure it is so easy to do > > > > > > 3. IMO this is completely transparent regarding where the > code > > > > lives, > > > > > > and "single CI" means a single build system that properly > manages > > > > the > > > > > > code > > > > > > to build > > > > > > > > > > > > Sorry if all the above was already clear and obvious to everyone > 😀 > > > > > > > > > > > > Best > > > > > > > > > > > > Gabriele > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno ven 10 mag 2024 alle ore 12:08 Enrique Gonzalez > Martinez < > > > > > > [email protected]> ha scritto: > > > > > > > > > > > > > Hi Alex > > > > > > > > > > > > > > You are mixing stuff here Alex. > > > > > > > > > > > > > > PR checks are there for guarding that nothing breaks when > somebody is > > > > > > > pushing code. If kie tools can break because is not there it > means we > > > > > > have > > > > > > > a problem in there. Keep in mind i am not talking about the > cause why > > > > > > they > > > > > > > are missing checks in our PR, it could be resources it could be > > > > > something > > > > > > > else. > > > > > > > > > > > > > > The other topic you are talking is the build. The question was > > > > > summarized > > > > > > > by Francisco.if we build the repos in order, does kie tool > able to > > > > > build > > > > > > > properly with the snapshots built in the pipeline or does need > > > > > additional > > > > > > > steps ? > > > > > > > > > > > > > > > > > > > > > Publishing artifacts is a whole different topic from the build. > > > > > Regarding > > > > > > > sync multiple deployments from the multirepo build i guess it > could > > > > be > > > > > a > > > > > > > problem, but also that is not the question here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > El vie, 10 may 2024, 11:55, Alex Porcelli <[email protected]> > > > > escribió: > > > > > > > > > > > > > > > Tiago already provided a in-depth explanation of the issue > in this > > > > > ML, > > > > > > > > that is summarized in the circular dependency thread [1] > > > > > > > > > > > > > > > > > > > > > > > > TL;DR: **Today** KIE-Tools is not part of any PR checks in > the > > > > > runtime > > > > > > > > repositories (drools, optaplanner, runtimes and apps) - so a > change > > > > > in > > > > > > > > those repos can break KIE Tools. If we were to continue to > use > > > > > > > > SNAPSHOTs, we need unify to a single CI that would include > all > > > > > > > > repositories - including KIE Tools. > > > > > > > > > > > > > > > > [1] - > > > > > https://lists.apache.org/thread/58xm7pqdyztf7qztmhvntf8wdmvfx7jx > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 5:15 AM Francisco Javier Tirado Sarti > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > Hi Tiago, > > > > > > > > > I would argue that, being all the repos part of the same > release, > > > > > > > mutable > > > > > > > > > snapshot is perfectly acceptable. As it currently happens > between > > > > > > > > Runtimes > > > > > > > > > repo and Drools repo (which have the Kie api) and Apps > repo and > > > > > > > Runtimes > > > > > > > > > repo. I, as I believe Enrique (at least if what I > understood from > > > > > his > > > > > > > > > e-mail) , do not understand why tooling should be > different in > > > > that > > > > > > > > regard. > > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 1:26 AM Tiago Bento < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Zanini and Enrique, > > > > > > > > > > > > > > > > > > > > Not sure if I'm making it sound differently, but > everything I'm > > > > > > > saying > > > > > > > > > > is also aiming towards a healthy CI. Not only for > SonataFlow or > > > > > > > Tools, > > > > > > > > > > though. For all of Apache KIE. > > > > > > > > > > > > > > > > > > > > My main point is that mutable Maven SNAPSHOTs are > harming us, > > > > > > giving > > > > > > > a > > > > > > > > > > false impression of synchronization between the repos. > It's an > > > > > > "easy > > > > > > > > > > way out" we've been historically abusing, IMHO. > > > > > > > > > > > > > > > > > > > > One of the best advantages of having multiple repos is > that > > > > they > > > > > > can > > > > > > > > > > be individually developed, tested, and managed. However, > by > > > > using > > > > > > > > > > Maven SNAPSHOTs as the synchronization mechanism, we > can't > > > > > > guarantee > > > > > > > > > > that, since repos become broken on any given day. More > than > > > > that, > > > > > > > it's > > > > > > > > > > impossible to go "back in time" on our development > branches, > > > > > since > > > > > > > the > > > > > > > > > > SNAPSHOT version will always point to latest. > > > > > > > > > > > > > > > > > > > > Please read this section from the guidelines of the > SciJava > > > > > project > > > > > > > at > > > > > > > > > > > > > > https://github.com/scijava/pom-scijava-base/blob/main/README.md. > > > > > > > > > > > > > > > > > > > > "Reproducible builds. This rule means no SNAPSHOT > dependencies, > > > > > no > > > > > > > > > > SNAPSHOT parents, and no SNAPSHOT plugin versions. A > snapshot > > > > > > version > > > > > > > > > > is not immutable, which means that code which depends on > a > > > > > snapshot > > > > > > > > > > may build today, but not build tomorrow, if the snapshot > is > > > > later > > > > > > > > > > changed. The best way to avoid this conundrum is to never > > > > depend > > > > > on > > > > > > > > > > SNAPSHOT versions. Snapshot are best used for testing > only; > > > > they > > > > > > can > > > > > > > > > > be used transiently, but their use should never make it > onto > > > > the > > > > > > main > > > > > > > > > > integration branch (e.g., main or master) of a project. > See > > > > also > > > > > > > Using > > > > > > > > > > snapshot couplings during development at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://imagej.net/develop/architecture#using-snapshot-couplings-during-development > > > > > > > > > > ." > > > > > > > > > > > > > > > > > > > > This summarizes very well the problem I'm trying to > convey > > > > here. > > > > > Of > > > > > > > > > > course, their words are much more clear than all the > > > > > explanations I > > > > > > > > > > tried giving so far. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2024 at 9:43 AM ricardo zanini fernandes > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > Tiago, > > > > > > > > > > > > > > > > > > > > > > Again, this is a major problem in our current > structure that > > > > we > > > > > > > > should > > > > > > > > > > use > > > > > > > > > > > another thread to discuss. For my proposal is to at > least > > > > have > > > > > a > > > > > > > > health > > > > > > > > > > > cloud build platform. We can completely remove UI from > our > > > > > images > > > > > > > in > > > > > > > > the > > > > > > > > > > > SonataFlow space and keep the consoles in kie-tools. > Once we > > > > > > solve > > > > > > > > the > > > > > > > > > > deep > > > > > > > > > > > structural problem we have, we can evaluate it better. > What > > > > we > > > > > > > can't > > > > > > > > do > > > > > > > > > > is > > > > > > > > > > > be behind and not release. > > > > > > > > > > > > > > > > > > > > > > The template image is pretty useful if you really want > to > > > > test > > > > > > the > > > > > > > > DevUI > > > > > > > > > > in > > > > > > > > > > > the image. You can easily pull the container file and > build > > > > the > > > > > > > image > > > > > > > > > > > during your integration tests and run the tests you > need. We > > > > > can > > > > > > do > > > > > > > > the > > > > > > > > > > > same in the image. I don't see a waste of resources, > but as a > > > > > > > > temporary > > > > > > > > > > > alternative until we figure out the deeper problem. > > > > > > > > > > > > > > > > > > > > > > At the moment, everything "tools" related is in a > single repo > > > > > and > > > > > > > the > > > > > > > > > > other > > > > > > > > > > > components are scattered in many other repos. We have > to > > > > decide > > > > > > if > > > > > > > we > > > > > > > > > > want > > > > > > > > > > > mono repos (infra-oriented) or context repos > > > > > (feature-oriented). > > > > > > We > > > > > > > > can't > > > > > > > > > > > have both otherwise the mono repos will start to drag > others > > > > to > > > > > > it, > > > > > > > > as we > > > > > > > > > > > are seeing happening. > > > > > > > > > > > > > > > > > > > > > > So IMO, we need first to sit down together and have a > new > > > > repo > > > > > > > > structure > > > > > > > > > > > design, perhaps based on components. We need to > evaluate. > > > > > > > > > > > > > > > > > > > > > > For the time being, we need to move forward at least > and I > > > > can > > > > > > live > > > > > > > > with > > > > > > > > > > > integration tests at the end of the line using final > > > > > > versions/built > > > > > > > > > > > versions of the DevUI if needed for SonataFlow. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2024 at 10:32 AM ricardo zanini > fernandes < > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Enrique! > > > > > > > > > > > > > > > > > > > > > > > > The images are using SNAPSHOTS from apps/runtimes > > > > currently. > > > > > > > There > > > > > > > > was > > > > > > > > > > a > > > > > > > > > > > > decision in the past to build apps in the images > from the > > > > > > > > > > main/versioned > > > > > > > > > > > > branch. We are changing this to use > SNAPSHOT/versioned > > > > > > artifacts > > > > > > > > > > published > > > > > > > > > > > > in Maven. About kie-tools, I don't know either. > > > > > > > > > > > > > > > > > > > > > > > > We can use immutable snapshots (weekly, nightlies) > instead. > > > > > I'm > > > > > > > not > > > > > > > > > > > > against it. But we do need those for integration > tests or > > > > at > > > > > > > least > > > > > > > > a > > > > > > > > > > robust > > > > > > > > > > > > CI platform to do integration without snapshots. > > > > > > > > > > > > > > > > > > > > > > > > Cheers! > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2024 at 5:07 AM Enrique Gonzalez > Martinez < > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Hi Tiago, > > > > > > > > > > > >> > > > > > > > > > > > >> I think the discussion is related to the graph > dependency > > > > > for > > > > > > > > now. So > > > > > > > > > > > >> I would keep that in mind. Anyway I am kinda > surprised > > > > about > > > > > > > some > > > > > > > > > > > >> concepts like (mutable snapshots and using snapshot > > > > > published > > > > > > > > > > > >> timestamps like they are final). > > > > > > > > > > > >> > > > > > > > > > > > >> Now I think I understand where the problem lies > now; so > > > > > > > basically > > > > > > > > if > > > > > > > > > > > >> we had the proper graph dependency project built > based on > > > > > > proper > > > > > > > > > > > >> snapshots everything would build just ok. So my > question > > > > is > > > > > > > pretty > > > > > > > > > > > >> simple, why is it not possible for kie tools or > kogito > > > > > images > > > > > > to > > > > > > > > use > > > > > > > > > > > >> the maven snapshot concept and point out to the last > > > > > snapshot > > > > > > > > build > > > > > > > > > > > >> (either local or remote) like other maven projects > work. > > > > Is > > > > > > > there > > > > > > > > any > > > > > > > > > > > >> limitation there ? > > > > > > > > > > > >> > > > > > > > > > > > >> El mié, 8 may 2024 a las 22:30, Tiago Bento (< > > > > > > > > [email protected]>) > > > > > > > > > > > >> escribió: > > > > > > > > > > > >> > > > > > > > > > > > > >> > Hi Zanini! Welcome back, hope you had a great time > > > > during > > > > > > your > > > > > > > > PTO. > > > > > > > > > > > >> > > > > > > > > > > > > >> > If I understand correctly, the way you see the > > > > integration > > > > > > > > between > > > > > > > > > > > >> > repos relies heavily on timestamped SNAPSHOTs. > This way: > > > > > > > > > > > >> > 1. `kogito-images` would stop building > `kogito-apps`, > > > > and > > > > > > > would > > > > > > > > rely > > > > > > > > > > > >> > on timestamped SNAPSHOTs being published from > > > > > > > > > > > >> > `drools/optaplanner/kogito-runtimes/kogito-apps`. > > > > > > > > > > > >> > 2. `kie-tools` would start publishing timestamped > > > > > SNAPSHOTs > > > > > > > for > > > > > > > > its > > > > > > > > > > UI > > > > > > > > > > > >> > components, so they would be consumed downstream > by > > > > > > > > > > > >> > `kogito-serverless-operator`. > > > > > > > > > > > >> > > > > > > > > > > > > >> > If that's what you mean, let me elaborate on some > of the > > > > > > > > > > consequences > > > > > > > > > > > >> > of this integration model, where we stop using the > > > > mutable > > > > > > > > > > > >> > 999-SNAPSHOT and start relying more on timestamped > > > > > SNAPSHOTs > > > > > > > for > > > > > > > > > > > >> > cross-repo synchronization: > > > > > > > > > > > >> > a. Cross-repo PRs would not be possible anymore, > > > > depending > > > > > > on > > > > > > > > which > > > > > > > > > > > >> > repos are involved in the ensemble. This would > create > > > > > > > boundaries > > > > > > > > > > > >> > between certain groups of our repos, namely: > > > > > > > > > > > >> > drools/optaplanner/kogito-runtimes/kogito-apps --> > > > > > > > > kogito-images --> > > > > > > > > > > > >> > kie-tools, --> kogito-serverless-operator. So 4 > > > > > "independent > > > > > > > > > > > >> > clusters". > > > > > > > > > > > >> > b. kogito-images, kie-tools, and > > > > > kogito-serverless-operator, > > > > > > > > using > > > > > > > > > > > >> > timestamped SNAPSHOTs between themselves, would > make the > > > > > > most > > > > > > > > > > upstream > > > > > > > > > > > >> > of the three (in this case, `kogito-images`), to > be the > > > > > one > > > > > > > > defining > > > > > > > > > > > >> > what timestamped SNAPSHOT version we should use > for its > > > > > > > upstream > > > > > > > > > > > >> > dependencies > > > > > > (drools/optaplanner/kogito-runtimes/kogito-apps). > > > > > > > > In > > > > > > > > > > more > > > > > > > > > > > >> > details: If `kogito-images` depends on > > > > > > > > > > > >> > `drools/optaplanner/kogito-runtimes/kogito-apps` > using > > > > > > version > > > > > > > > > > > >> > 999-20240501, that's the version `kie-tools` and > > > > > > > > > > > >> > `kogito-serverless-operator` need to use as well, > as the > > > > > > > > > > > >> > drools/optaplanner/kogito-runtimes/kogito-apps > would > > > > come > > > > > > > > > > transitively > > > > > > > > > > > >> > from the `kogito-images` dependency these two > repos > > > > would > > > > > > > have. > > > > > > > > > > This, > > > > > > > > > > > >> > IMHO, causes a lot of confusion, since the dates > on the > > > > > > > > timestamped > > > > > > > > > > > >> > SNAPSHOT versions have implicit transitive > dependencies > > > > to > > > > > > > other > > > > > > > > > > > >> > upstream projects that also have their own > timestamped > > > > > > > SNAPSHOTs > > > > > > > > > > > >> > published. I.e., 999-20240501 of `kie-tools` > won't be > > > > > > aligned > > > > > > > > with > > > > > > > > > > > >> > 999-2024051 of > > > > > > > `drools/optaplanner/kogito-runtimes/kogito-apps`. > > > > > > > > > > > >> > c. Updating dependencies across the board would > be much > > > > > more > > > > > > > > > > > >> > difficult, as we would need to rely on a chain > reaction > > > > > > > starting > > > > > > > > > > from > > > > > > > > > > > >> > the > `drools/optaplanner/kogito-runtimes/kogito-apps` > > > > > cluster > > > > > > > > all the > > > > > > > > > > > >> > way down to `kogito-serverless-operator`, > releasing > > > > > > > timestamped > > > > > > > > > > > >> > SNAPSHOTs and sending PRs for each "cluster" > along the > > > > > way. > > > > > > > > > > > >> > > > > > > > > > > > > >> > I'm not sure this is the way we want to move > forward, to > > > > > be > > > > > > > > honest, > > > > > > > > > > as > > > > > > > > > > > >> > these consequences need to be thoroughly examined > by > > > > > people > > > > > > in > > > > > > > > the > > > > > > > > > > > >> > community, so that all the side-effects of this > choice > > > > are > > > > > > > > clear to > > > > > > > > > > > >> > everyone. > > > > > > > > > > > >> > > > > > > > > > > > > >> > To conclude, having `kie-tools` building a > "template > > > > > image" > > > > > > to > > > > > > > > > > verify > > > > > > > > > > > >> > that it will not break > `kogito-serverless-operator` > > > > > > > downstream, > > > > > > > > and > > > > > > > > > > > >> > having `kogito-serverless-operator` building > parts of > > > > > > > > `kie-tools` to > > > > > > > > > > > >> > incorporate the latest changes, IMHO, is: > > > > > > > > > > > >> > 1. Duplication of building logic. IMHO, we can't > have > > > > > > > individual > > > > > > > > > > repos > > > > > > > > > > > >> > creating its own up- and downstream CI systems. > > > > > > > > > > > >> > 2. Conflicting with the timestamped SNAPSHOTs > strategy > > > > for > > > > > > > > > > integrating > > > > > > > > > > > >> > repos, where boundaries are clearly defined. > > > > > > > > > > > >> > 3. Wasteful use of resources (same that is > happening > > > > with > > > > > > > > > > > >> > `kogito-images` building `kogito-apps` right now). > > > > > > > > > > > >> > > > > > > > > > > > > >> > I'm sorry for bringing all this to this seemingly > simple > > > > > > > > proposal, > > > > > > > > > > but > > > > > > > > > > > >> > I'm afraid taking steps without knowing where > we're > > > > going > > > > > > > isn't > > > > > > > > the > > > > > > > > > > > >> > way to move forward. > > > > > > > > > > > >> > > > > > > > > > > > > >> > Please let me know if I misunderstood something. > > > > > > > > > > > >> > > > > > > > > > > > > >> > On Wed, May 8, 2024 at 3:15 PM ricardo zanini > fernandes > > > > > > > > > > > >> > <[email protected]> wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Hi! > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Sorry for the late reply, I was on PTO. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Replies inline. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > The `kie-tools` CI is not prepared to build > > > > > > > > > > > >> > > `kogito-serverless-operator`, so the way > > > > > > > > > > `kogito-serverless-operator` > > > > > > > > > > > >> > > references the SonataFlow Quarkus Dev UI will be > > > > > important > > > > > > > to > > > > > > > > > > > >> > > establish the boundaries between both repos. To > > > > further > > > > > > > > develop > > > > > > > > > > the > > > > > > > > > > > >> > > SonataFlow Quarkus Dev UI and have its changes > reflect > > > > > on > > > > > > > the > > > > > > > > > > > >> > > `kogito-swf-devmode` image, we need to come up > with a > > > > > > > strategy > > > > > > > > > > that is > > > > > > > > > > > >> > > both safe, consistent, and enforces correctness. > > > > There's > > > > > > > also > > > > > > > > the > > > > > > > > > > fact > > > > > > > > > > > >> > > that currently `kie-tools` depends on > timestamped > > > > > > SNAPSHOTs > > > > > > > of > > > > > > > > > > > >> > > Kogito/Drools, while > `kogito-serverless-operator` uses > > > > > > > > > > 999-SNAPSHOT, > > > > > > > > > > > >> > > if I'm not mistaken. Can you elaborate a little > bit > > > > more > > > > > > on > > > > > > > > how > > > > > > > > > > you > > > > > > > > > > > >> > > see this reference being done? Please consider > > > > > cross-repo > > > > > > > PRs > > > > > > > > for > > > > > > > > > > big > > > > > > > > > > > >> > > migrations like the Quarkus 3.8.4 that is > currently > > > > > > > happening. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > This will make us discuss yet again the need of > > > > > snapshots > > > > > > > for > > > > > > > > the > > > > > > > > > > UI > > > > > > > > > > > >> > > components used by the images. The images must > be the > > > > > end > > > > > > of > > > > > > > > the > > > > > > > > > > build > > > > > > > > > > > >> > > pipeline, where we aggregate every component we > ship > > > > in > > > > > a > > > > > > > > single > > > > > > > > > > > >> instance > > > > > > > > > > > >> > > to release. Yet, they are a core part of the > cloud > > > > > > platform, > > > > > > > > hence > > > > > > > > > > > >> part of > > > > > > > > > > > >> > > the Operator BDD, testing, integration, and > > > > delivering. > > > > > > > > > > Intermediate > > > > > > > > > > > >> repos > > > > > > > > > > > >> > > can't depend on them unless they are also > responsible > > > > > for > > > > > > > > > > maintenance > > > > > > > > > > > >> and > > > > > > > > > > > >> > > release like what we're doing with the consoles. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Also, the images can have a respin anytime we > need a > > > > new > > > > > > > > component > > > > > > > > > > > >> bump. We > > > > > > > > > > > >> > > do this all the time to fix CVEs or bug fixes > in a > > > > > > specific > > > > > > > > > > component > > > > > > > > > > > >> that > > > > > > > > > > > >> > > is part of the image. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > The devui can be developed and maintained > within the > > > > > > > kie-tools > > > > > > > > > > repo > > > > > > > > > > > >> and > > > > > > > > > > > >> > > have tests that verify that component. The > integration > > > > > > will > > > > > > > be > > > > > > > > > > made > > > > > > > > > > > >> on the > > > > > > > > > > > >> > > Images/Operator side once we grab a latest > snapshot. > > > > > > > > > > Alternatively, > > > > > > > > > > > >> we can > > > > > > > > > > > >> > > do: > > > > > > > > > > > >> > > 1. The tools CI can fetch a template image, > inject the > > > > > > > > component, > > > > > > > > > > and > > > > > > > > > > > >> run > > > > > > > > > > > >> > > the required tests > > > > > > > > > > > >> > > 2. The image/operator CI can fetch and build > the UI > > > > > > > > components and > > > > > > > > > > > >> run the > > > > > > > > > > > >> > > integration tests > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Also, regarding point "4. Review the PR GHA > checks > > > > to > > > > > > run > > > > > > > > CLI > > > > > > > > > > tests > > > > > > > > > > > >> > > once there's a change in the cmd module" of the > > > > proposed > > > > > > > > EPIC, I > > > > > > > > > > think > > > > > > > > > > > >> > > we might run into problems, since the `cmd` > module > > > > also > > > > > > > > depends > > > > > > > > > > on the > > > > > > > > > > > >> > > `api` and `workflowproj` modules of > > > > > > > > `kogito-serverless-operator.` > > > > > > > > > > I'm > > > > > > > > > > > >> > > afraid changes made to these two modules would > also > > > > need > > > > > > to > > > > > > > > > > trigger a > > > > > > > > > > > >> > > build of the `cmd` module, and they can > potentially > > > > > break > > > > > > > it. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Sorry to not get into details there, but the > new cmd > > > > > > module > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > >> part of > > > > > > > > > > > >> > > the Operator's workspace, any changes in any > cross > > > > > > > > dependencies > > > > > > > > > > there > > > > > > > > > > > >> will > > > > > > > > > > > >> > > trigger the CI checks, as we are doing today in > the > > > > > > builder > > > > > > > > and > > > > > > > > > > > >> > > workflowproj modules. It will be way easier to > > > > maintain > > > > > > the > > > > > > > > CLI. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > These considerations alone, IMHO, expose one > of the > > > > > > > biggest > > > > > > > > > > > >> challenges > > > > > > > > > > > >> > > we have in our community right now, which is > that the > > > > > > > > definition > > > > > > > > > > and > > > > > > > > > > > >> > > implementation of the dependency graph between > > > > > > > > > > repos/modules/packages > > > > > > > > > > > >> > > is currently spread across many different "build > > > > > systems", > > > > > > > > like > > > > > > > > > > the > > > > > > > > > > > >> > > new proposed GHA jobs exclusive to the > > > > > > > > > > `kogito-serverless-operator` > > > > > > > > > > > >> > > repo, the Build Chain system we have for the > > > > > Drools/Kogito > > > > > > > > repos, > > > > > > > > > > the > > > > > > > > > > > >> > > `kie-tools` CI, and the many Jenkins jobs we > have on > > > > > > > Apache's > > > > > > > > > > Jenkins. > > > > > > > > > > > >> > > There's also the fact that we have > `kogito-images` > > > > > > > selectively > > > > > > > > > > > >> > > building parts of `kogito-apps` during its own > build > > > > to > > > > > > > > include > > > > > > > > > > them > > > > > > > > > > > >> > > in the images it produces. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > I agree. We would need to sit together and > solve this > > > > > as a > > > > > > > > team, > > > > > > > > > > > >> having a > > > > > > > > > > > >> > > nice integration across every repo. The > kogito-images > > > > > > > building > > > > > > > > > > parts > > > > > > > > > > > >> of the > > > > > > > > > > > >> > > apps is something we currently working on, > IIRC. The > > > > > apps > > > > > > > will > > > > > > > > > > deploy > > > > > > > > > > > >> a > > > > > > > > > > > >> > > snapshot as part of the nightlies and the > images will > > > > be > > > > > > > > using it. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > I just think it is important to highlight > that this > > > > > > > proposal > > > > > > > > > > would > > > > > > > > > > > >> > > only address a LOCAL problem related > exclusively to > > > > the > > > > > > > > SonataFlow > > > > > > > > > > > >> > > section of the Apache KIE community, while, at > the > > > > same > > > > > > > time, > > > > > > > > not > > > > > > > > > > > >> > > contributing to reducing the segmentation of the > > > > Apache > > > > > > KIE > > > > > > > > > > community > > > > > > > > > > > >> > > as a whole. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Yes, the proposal is to starting addressing the > local > > > > > > Apache > > > > > > > > KIE > > > > > > > > > > > >> SonataFlow > > > > > > > > > > > >> > > Cloud platform first. But I agree that we need > to > > > > > refactor > > > > > > > > our CI > > > > > > > > > > as a > > > > > > > > > > > >> > > whole, which is something we should start in a > new > > > > > thread. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Cheers! > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On Mon, Apr 22, 2024 at 10:02 PM Tiago Bento < > > > > > > > > > > [email protected]> > > > > > > > > > > > >> wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Zanini and Alex, > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > The task we agreed on for after releasing > Apache KIE > > > > > 10 > > > > > > is > > > > > > > > > > > >> > > > > > > > > > > https://github.com/apache/incubator-kie-issues/issues/1040. > > > > > > > > It > > > > > > > > > > only > > > > > > > > > > > >> > > > describes deleting the temporary copies we'll > have > > > > on > > > > > > KIE > > > > > > > > Tools > > > > > > > > > > and > > > > > > > > > > > >> > > > reverting things back to where they were, > using the > > > > > > fixed > > > > > > > > 10.0.0 > > > > > > > > > > > >> > > > version. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Moving `kn-plugin-workflow` and the > > > > > > > > > > `kogito-swf-{devmode,builder}` > > > > > > > > > > > >> > > > images to `kogito-serverless-operator` would > be a > > > > new > > > > > > > move, > > > > > > > > > > which I > > > > > > > > > > > >> > > > understand is the scope of this proposal > thread. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Doing so, however, will make > > > > > > `kogito-serverless-operator` > > > > > > > > > > depend on > > > > > > > > > > > >> > > > `kie-tools`, since the SonataFlow Quarkus Dev > UI is > > > > > > hosted > > > > > > > > there > > > > > > > > > > > >> now, > > > > > > > > > > > >> > > > and it is a dependency of the > `kogito-swf-devmode` > > > > > > image. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > I'm saying this because I think we need to > further > > > > > > discuss > > > > > > > > the > > > > > > > > > > > >> > > > consequences of this change... > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > The `kie-tools` CI is not prepared to build > > > > > > > > > > > >> > > > `kogito-serverless-operator`, so the way > > > > > > > > > > > >> `kogito-serverless-operator` > > > > > > > > > > > >> > > > references the SonataFlow Quarkus Dev UI will > be > > > > > > important > > > > > > > > to > > > > > > > > > > > >> > > > establish the boundaries between both repos. > To > > > > > further > > > > > > > > develop > > > > > > > > > > the > > > > > > > > > > > >> > > > SonataFlow Quarkus Dev UI and have its changes > > > > reflect > > > > > > on > > > > > > > > the > > > > > > > > > > > >> > > > `kogito-swf-devmode` image, we need to come > up with > > > > a > > > > > > > > strategy > > > > > > > > > > that > > > > > > > > > > > >> is > > > > > > > > > > > >> > > > both safe, consistent, and enforces > correctness. > > > > > There's > > > > > > > > also > > > > > > > > > > the > > > > > > > > > > > >> fact > > > > > > > > > > > >> > > > that currently `kie-tools` depends on > timestamped > > > > > > > SNAPSHOTs > > > > > > > > of > > > > > > > > > > > >> > > > Kogito/Drools, while > `kogito-serverless-operator` > > > > uses > > > > > > > > > > 999-SNAPSHOT, > > > > > > > > > > > >> > > > if I'm not mistaken. Can you elaborate a > little bit > > > > > more > > > > > > > on > > > > > > > > how > > > > > > > > > > you > > > > > > > > > > > >> > > > see this reference being done? Please consider > > > > > > cross-repo > > > > > > > > PRs > > > > > > > > > > for > > > > > > > > > > > >> big > > > > > > > > > > > >> > > > migrations like the Quarkus 3.8.4 that is > currently > > > > > > > > happening. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Also, regarding point "4. Review the PR GHA > checks > > > > to > > > > > > run > > > > > > > > CLI > > > > > > > > > > tests > > > > > > > > > > > >> > > > once there's a change in the cmd module" of > the > > > > > proposed > > > > > > > > EPIC, I > > > > > > > > > > > >> think > > > > > > > > > > > >> > > > we might run into problems, since the `cmd` > module > > > > > also > > > > > > > > depends > > > > > > > > > > on > > > > > > > > > > > >> the > > > > > > > > > > > >> > > > `api` and `workflowproj` modules of > > > > > > > > > > `kogito-serverless-operator.` > > > > > > > > > > > >> I'm > > > > > > > > > > > >> > > > afraid changes made to these two modules > would also > > > > > need > > > > > > > to > > > > > > > > > > trigger > > > > > > > > > > > >> a > > > > > > > > > > > >> > > > build of the `cmd` module, and they can > potentially > > > > > > break > > > > > > > > it. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > These considerations alone, IMHO, expose one > of the > > > > > > > biggest > > > > > > > > > > > >> challenges > > > > > > > > > > > >> > > > we have in our community right now, which is > that > > > > the > > > > > > > > > > definition and > > > > > > > > > > > >> > > > implementation of the dependency graph between > > > > > > > > > > > >> repos/modules/packages > > > > > > > > > > > >> > > > is currently spread across many different > "build > > > > > > systems", > > > > > > > > like > > > > > > > > > > the > > > > > > > > > > > >> > > > new proposed GHA jobs exclusive to the > > > > > > > > > > `kogito-serverless-operator` > > > > > > > > > > > >> > > > repo, the Build Chain system we have for the > > > > > > Drools/Kogito > > > > > > > > > > repos, > > > > > > > > > > > >> the > > > > > > > > > > > >> > > > `kie-tools` CI, and the many Jenkins jobs we > have on > > > > > > > > Apache's > > > > > > > > > > > >> Jenkins. > > > > > > > > > > > >> > > > There's also the fact that we have > `kogito-images` > > > > > > > > selectively > > > > > > > > > > > >> > > > building parts of `kogito-apps` during its > own build > > > > > to > > > > > > > > include > > > > > > > > > > them > > > > > > > > > > > >> > > > in the images it produces. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > With all that said, I'm not opposed to moving > the > > > > > > > > > > > >> `kn-workflow-plugin` > > > > > > > > > > > >> > > > package from `kie-tools` to the `cmd` module > of > > > > > > > > > > > >> > > > `kogito-serverless-operator`. In fact, like I > said > > > > in > > > > > > the > > > > > > > > past, > > > > > > > > > > I > > > > > > > > > > > >> > > > think it makes a lot of sense that they're > part of > > > > the > > > > > > > same > > > > > > > > > > > >> > > > compilation unit, as they're very closely > related, > > > > and > > > > > > > > should > > > > > > > > > > > >> > > > therefore be in sync at all times. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > I just think it is important to highlight > that this > > > > > > > proposal > > > > > > > > > > would > > > > > > > > > > > >> > > > only address a LOCAL problem related > exclusively to > > > > > the > > > > > > > > > > SonataFlow > > > > > > > > > > > >> > > > section of the Apache KIE community, while, > at the > > > > > same > > > > > > > > time, > > > > > > > > > > not > > > > > > > > > > > >> > > > contributing to reducing the segmentation of > the > > > > > Apache > > > > > > > KIE > > > > > > > > > > > >> community > > > > > > > > > > > >> > > > as a whole. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Regards, > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Tiago Bento > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > On Fri, Apr 19, 2024 at 4:08 PM ricardo zanini > > > > > fernandes > > > > > > > > > > > >> > > > <[email protected]> wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > Alex, > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > Yes, in the proposal we just barely > outlined. I > > > > > create > > > > > > > the > > > > > > > > > > EPIC > > > > > > > > > > > >> to have > > > > > > > > > > > >> > > > > more details and start working on them as > soon as > > > > we > > > > > > > > agree. > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > On Fri, Apr 19, 2024 at 4:24 PM Alex > Porcelli < > > > > > > > > > > [email protected]> > > > > > > > > > > > >> wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > Thank you for outlining the tasks post > the 10.x > > > > > > > release. > > > > > > > > > > It's > > > > > > > > > > > >> > > > > > important to note that these are already > > > > included > > > > > in > > > > > > > the > > > > > > > > > > amended > > > > > > > > > > > >> > > > > > proposal [1], specifically in steps 9 and > 10, > > > > > which > > > > > > > the > > > > > > > > > > > >> community has > > > > > > > > > > > >> > > > > > voted on. If there are concerns about the > > > > > execution > > > > > > of > > > > > > > > these > > > > > > > > > > > >> steps, > > > > > > > > > > > >> > > > > > I'd like to reassure you that they will > proceed > > > > as > > > > > > > > planned, > > > > > > > > > > in > > > > > > > > > > > >> > > > > > compliance with Apache guidelines. > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > Looking ahead, after the release of > version 10, > > > > we > > > > > > > > already > > > > > > > > > > > >> agreed that > > > > > > > > > > > >> > > > > > we'll need to have a thorough discussion > > > > regarding > > > > > > the > > > > > > > > > > codebase > > > > > > > > > > > >> > > > > > structure. This will allow us to refine > our > > > > > > > > understanding > > > > > > > > > > of the > > > > > > > > > > > >> > > > > > sub-brands, their interrelationships, and > their > > > > > > > > strategic > > > > > > > > > > > >> positioning. > > > > > > > > > > > >> > > > > > I agree that this is crucial for our next > steps > > > > > and > > > > > > > look > > > > > > > > > > > >> forward to > > > > > > > > > > > >> > > > > > our collaborative efforts in shaping this. > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > [1] - > > > > > > > > > > > >> > > > > > > > > https://lists.apache.org/thread/pw327lkpmy9gxklq4t5zbwzxjo2y3c0w > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > On Fri, Apr 19, 2024 at 2:50 PM ricardo > zanini > > > > > > > fernandes > > > > > > > > > > > >> > > > > > <[email protected]> wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > Folks, > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > I've outlined the tasks we need once we > > > > release > > > > > > 10.x > > > > > > > > from > > > > > > > > > > > >> kie-tools: > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-issues/issues/1102 > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > Once we release, we can detail this > planning > > > > and > > > > > > > start > > > > > > > > > > > >> working on it > > > > > > > > > > > >> > > > to > > > > > > > > > > > >> > > > > > > have a streamlined process for the next > > > > release. > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > Please let me know if it makes sense to > you. > > > > We > > > > > > can > > > > > > > > break > > > > > > > > > > > >> down and > > > > > > > > > > > >> > > > detail > > > > > > > > > > > >> > > > > > > the tasks once we agree on this initial > plan. > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > Cheers! > > > > > > > > > > > >> > > > > > > -- > > > > > > > > > > > >> > > > > > > Ricardo Zanini Fernandes > > > > > > > > > > > >> > > > > > > Vida longa e próspera. > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > >> > > > > > To unsubscribe, e-mail: > > > > > > > [email protected] > > > > > > > > > > > >> > > > > > For additional commands, e-mail: > > > > > > > > [email protected] > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > >> > > > To unsubscribe, e-mail: > > > > > [email protected] > > > > > > > > > > > >> > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > >> > To unsubscribe, e-mail: > [email protected] > > > > > > > > > > > >> > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > >> To unsubscribe, e-mail: > [email protected] > > > > > > > > > > > >> For additional commands, e-mail: > [email protected] > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
