Team, I've created this placeholder GitHub Issue [1] so we can track and follow progress on this discussion.
[1] - https://github.com/apache/incubator-kie-issues/issues/706 On Fri, Nov 17, 2023 at 6:56 AM Enrique Gonzalez Martinez <[email protected]> wrote: > > So far there is no poc or anything. We just throw the idea so if we agree > on the concept of what is needed in order to achieve simplified architecture > > El vie, 17 nov 2023, 12:52, Walter Medvedeo <[email protected]> escribió: > > > Ok, nice, It will be good to see all this added into the jobs service. If > > you have guys any design document, or prototype you can share at some > > point, it'll be great. > > > > Thanks, > > Walter. > > > > > > On 2023/11/17 10:19:35 Enrique Gonzalez Martinez wrote: > > > 1. There are not distributed tx... There are several tx. > > > 2. V7 has this problem regarding the singleton service. As we relied on > > > eap/wildfly. This one has an interesting concept called timer partition > > so > > > u can split in fact a singleton service in a cluster environment without > > > colliding or multiplying the trigger if that is u worry about. Also there > > > are a ways to create locks through the state so no other instances will > > > clash when they are manipulating the jobs. > > > > > > El vie, 17 nov 2023, 10:04, Walter Medvedeo <[email protected]> > > escribió: > > > > > > > Hi Enrique, > > > > so according with 2) you will provide two transport types? > > > > 2.1) local access via Pojos > > > > 2.2) rest client invoking the job service in a distributed tx > > > > > > > > Finally, on last thing that is important to consider is that the jobs > > > > service is currently a singleton service, so, depending on how much of > > the > > > > logic of the job service you'll execute in the kogito.-runtime side, > > this > > > > might introduce an additional challenge. > > > > > > > > Regards, > > > > Walter. > > > > > > > > On 2023/11/16 16:14:06 Enrique Gonzalez Martinez wrote: > > > > > HI Walter: > > > > > > > > > > Not sure if I understood properly what you are assessing here. > > > > > > > > > > The idea is to have a > > > > > 1) core component (let's say an interface job service) that is > > > > > responsible of the logic of the job service (including storage) > > > > > 2) two different types of transport (in vm) deployed within the > > > > > runtime and just passing pojo from the runtime to the job service and > > > > > a different as a rest client invoking the job service in a > > distributed > > > > tx. > > > > > > > > > > It does not really matter where the transaction is executed because > > > > > the only difference is the transport. Forget for an instance the > > > > > reactive stuff for now. > > > > > > > > > > if you have an in-vm deployed / colocated / embedded (same jvm) and > > > > > there is an overdue trigger... the lifecycle will still be the same > > > > > (scheduled - completed) but the entire thing will happen in the same > > > > > tx and jvm. > > > > > > > > > > if you have in a different subsystem (2 jvm) what will happen is that > > > > > the way you send the message will should be involved in the tx so > > what > > > > > will you have is > > > > > > > > > > # runtime > > > > > executed some stuff > > > > > create job service > > > > > -send message to job service > > > > > > > > > > # job service > > > > > > > > > > the other jvm > > > > > consume message > > > > > create job > > > > > schedule for execution > > > > > > > > > > that will be the other tx in the second jvm > > > > > > > > > > > > > > > that is what we want to achieve. There might be cases where reactive > > > > > might work out or not, we should design this properly. > > > > > > > > > > > > > > > It is not possible to make all combinations available to the end user > > > > > (and it is not even needed - it won't add any value) > > > > > > > > > > > > > > > > > > > > El jue, 16 nov 2023 a las 15:32, Walter Medvedeo > > > > > (<[email protected]>) escribió: > > > > > > > > > > > > Maybe I'm overthinking, or having a technical gap :(, but what I > > don't > > > > see right now guys is how to solve this challenge. (in case of option2) > > > > > > > > > > > > Let me explain in a very simplified way. At some point in time we > > need > > > > to persist the runtime info and the jobs service info in single write. > > > > > > > > > > > > Scenario1: (single write) > > > > > > > > > > > > 1) OpenTransaction > > > > > > > > > > > > 2) Write process the process information with a non-reactive > > > > datasource. > > > > > > 3) Write the job-service related information to create a job, > > this > > > > time we use not only a different datasource, but also a > > reactive-datasource. > > > > > > > > > > > > 4) Commit Transaction > > > > > > > > > > > > What I don't see is how this will work. > > > > > > > > > > > > > > > > > > Scenario2: (overdue trigger) > > > > > > > > > > > > 1) OpenTransaction > > > > > > > > > > > > 2) Write process the process information with a non-reactive > > > > datasource. > > > > > > 3) Write the job-service related information to create a job, > > this > > > > time we use not only a different datasource, but also a > > reactive-datasource. > > > > > > 3.5) It's detected that the job is overdue and must be > > executed, > > > > let's execute it. > > > > > > > > > > > > 4) Commit Transaction > > > > > > 5) (another posibility instead of 3.5) > > > > > > After the commit, it's detected that the job is overdue and must be > > > > executed, let's execute it. > > > > > > > > > > > > Now, where do the code to achieve the execution of 3.5, or 5 > > execute? > > > > in the kogito-runtimes JVM or in the jobs-service JVM? > > > > > > > > > > > > Regards, > > > > > > Walter. > > > > > > > > > > > > > > > > > > On 2023/11/16 10:38:13 Enrique Gonzalez Martinez wrote: > > > > > > > Hi Walter: > > > > > > > > > > > > > > 1) you can still be reactive if you like if you build a transport > > > > > > > layer being reactive. so moving that to the transport layer you > > will > > > > > > > have the same benefits not blocking the IO at the endpoint level > > IMO. > > > > > > > In any case using this sort of approach. > > > > > > > 2) that should not invalidate the logic. It is to say. The job > > > > service > > > > > > > will the the create - schedule - fire execution in place. > > > > > > > > > > > > > > Ideally this should be like > > > > > > > > > > > > > > as colocated service > > > > > > > runtime -> job service API -> embedded job service proxy (in vm > > > > > > > transport) -> job service > > > > > > > > > > > > > > as distributed > > > > > > > runtime -> job service API -> rest job service proxy (rest > > client) || > > > > > > > wire || -> endpoint (reactive or not) -> job service > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > El jue, 16 nov 2023 a las 10:15, Walter Medvedeo > > > > > > > (<[email protected]>) escribió: > > > > > > > > > > > > > > > > Hi Alex, and guys, thanks for starting discussion. > > > > > > > > > > > > > > > > Let me add some comments that I think might be useful to > > consider > > > > for the evaluation/implementation. > > > > > > > > > > > > > > > > Regarding option 2) we must keep in mind that right now, the > > jobs > > > > service DB writes are reactive, as well as the scheduling process, > > etc. If > > > > we keep this approach, I think we have a potential challenge here on > > how to > > > > integrate the non reactive kogito-runtime DB writes (and it's > > synchronous > > > > execution model in general) with the subsequent reactive schedule of > > the > > > > job. Since in the end, we want all this to happen in the same > > transaction. > > > > > > > > What are the plan in that regard? > > > > > > > > > > > > > > > > > > > > > > > > Another thing to keep in mind is the treatment of overdue jobs. > > > > There are situations where the job being created has already > > overpassed the > > > > execution time, in these cases, it's automatically fired, and the > > > > corresponding associated action is executed immediatelly. If I don't > > recall > > > > wrong, this is decided as part of the scheduling process. I think we > > must > > > > be sure that we can keep the possibility to fire ouverdue jobs > > (something > > > > that is optional configurable) at the same time we don't want the > > execution > > > > of the job is produced in the kogito-runtimes side but in the > > jobs-service. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Walter. > > > > > > > > > > > > > > > > > > > > > > > > On 2023/11/15 20:58:11 Alex Porcelli wrote: > > > > > > > > > Similar to Enrique, I'd +1 the second option.. as it aligns > > > > better > > > > > > > > > with the current data-index approach. It would allow, for > > sake of > > > > > > > > > simplicity. > > > > > > > > > > > > > > > > > > Are we ok if we move forward with the second option as a > > > > proposal to > > > > > > > > > be implemented? > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 14, 2023 at 4:40 AM Enrique Gonzalez Martinez > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > That case is fixed as it uses the EventPublisher interface. > > > > > > > > > > > > > > > > > > > > El mar, 14 nov 2023, 10:25, Pere Fernandez (apache) < > > > > [email protected]> > > > > > > > > > > escribió: > > > > > > > > > > > > > > > > > > > > > Something we may also need to think about is the > > > > communication between the > > > > > > > > > > > Job-Service and the Data-Index. Currently it Job-Service > > > > needs to message > > > > > > > > > > > the Data-Index when a Job status changes. I think that > > for > > > > the purpose of > > > > > > > > > > > the Simplified Architecture it should be able to write > > > > directly into the > > > > > > > > > > > Data-Index DB, in a similar manner as the runtime does > > when > > > > the > > > > > > > > > > > `kogito-addons-quarkus-data-index-persistence` addons are > > > > present. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
