I moved the issue to kie-isses and completely rephrase it with a summary of our discussion https://github.com/apache/incubator-kie-issues/issues/1181
On Wed, May 8, 2024 at 5:21 PM Enrique Gonzalez Martinez < egonza...@apache.org> wrote: > Hi Francisco, > > I am ok as well to set the variable a way to read / write the var > value as there is also an idiom for BPMN (DataObject with state) > The downside of this solution is how to design the sync - up, as we > are going to need somehow to add state to the variable (at least > loaded, dirty write, ... somehow)... this is something we can add to > the Variable itself. In the store I did mention before it is clear > that you use and you forget once it is used (no need for states) in > the case of this is a bit more difficult. > The only constraint for the design I would add is that the persistence > is something decoupled from the variable itself for the case we can > reuse that in other future features. > > I am ok with both approaches. > > PS: Regarding the store node. BPMN needs some support for this sort of > scenario so we will need to include a way to support this in the > future. It is not possible to implement this with the current > connection types and activity input/output calculations. So that would > need to be changed somehow. but that is some other discussion > > > > > El mié, 8 may 2024 a las 15:26, Francisco Javier Tirado Sarti > (<ftira...@redhat.com>) escribió: > > > > I think we need a "kogito context variable" interface to read from/write > > to, implemented with the constraints you mentioned. > > I'm not against adding that new node (which should be using that API) you > > propose, but it won't be useful for SWF case (In my opinion, there is not > > need to add a new node to the process instance execution list every time > > you want to read/write from the context) > > IN SWF we would add a reserved keyword $WORKLOW.context.<variable name> > to > > read/write to the context (using jq expressions) that directly invoke the > > "kogito context variable" interface underneath (as part of the expression > > evaluation) > > > > > > > > > > On Tue, May 7, 2024 at 7:13 PM Enrique Gonzalez Martinez < > > egonza...@apache.org> wrote: > > > > > Ok, so les brief this a bit. > > > The constraints i would set are: > > > 1. Should be stored outside the process state > > > 2. Pluggable (data base store as default impl) > > > 3. Should be lazy load. Not required to access anytime we access the > > > process state but should be involved automatically in any input/output > like > > > a process or context variable > > > 4. Setup should be logically defined through idiom definition but > wired at > > > runtime level. It is to say. The workflow definition should say this > store > > > points to this logica store and runtime define the mappimg between the > > > logical store and physical storage. > > > 5. The store should define what is stored in it. For instance it has > store > > > the variable name with this structure. ( that way we can use those > > > identifiers like variables amd we can compute the outcome type) > > > > > > I would suggest an impl like this > > > > > > 1. Create a new type of node. Let's call it store > > > 2. As this node is special type we need a way to tell the engine this > will > > > be part of input/ output. For this we can create a new type of > connection ( > > > different from engine path execution- DROOLS_CONNECTION.... > DROOLS_STORE or > > > something like that) > > > 3. Once we reach an activity we look for these connections and add > them to > > > the inputs/outputs > > > 4. For inputs we call load operation. For outputs we call save. > > > 5. For computing how to load something we have compose the context. For > > > trivial cases we can use process instance id + variable name. For > cluster > > > operation we can use something instan es share among them.the important > > > thing is that process need to know how to compose that context. > > > > > > > > > In the case of BPMN we have the data store for this. Dont know if > therecis > > > something different for swf. > > > > > > That would be an initial idea. > > > > > > El mar, 7 may 2024, 17:57, Francisco Javier Tirado Sarti < > > > ftira...@redhat.com> escribió: > > > > > > > Hi Enrique, > > > > Of course I'm open to exploring that path. > > > > I guess we can have a table where we store the context variables and > that > > > > table will be read every time we execute an instance that is reading > > > > information from context. > > > > There are some questions that we will need to answer. > > > > How do we populate that context? Through another workflow? Can the > > > context > > > > be arbitrarily written? > > > > Which will be the context key? Just the property name (assuming it > will > > > be > > > > unique across the application)? The property name plus a context id > (to > > > > allow reusing the same name for differente domains)? > > > > I let you propose the BPMN idiom and we can think of something > equivalent > > > > for SWF. > > > > Thanks a lot. > > > > > > > > > > > > > > > > On Tue, May 7, 2024 at 5:27 PM Enrique Gonzalez Martinez < > > > > egonza...@apache.org> wrote: > > > > > > > > > Hi Francisco, > > > > > Regarding your notion of context. We had in the past something > called > > > > > globals. This was a store shared among processes usually through > kie > > > > > session. Actually not long ago i had to made a hack to support > globals > > > at > > > > > instance level. > > > > > In this case i am more open to discuss but shifting a bit the > > > discussion > > > > to > > > > > something more general if u dont mind. > > > > > > > > > > Having stores shared among processes is not a bad idea and we can > make > > > it > > > > > work for process instance cooperation. > > > > > > > > > > Maybe we can create variables in the process that are stored > outside > > > the > > > > > process state. > > > > > > > > > > In this case I think this is more sensible than using human task > as an > > > > > event. > > > > > > > > > > In this case there are several options and does not affect the > engine > > > at > > > > > the very core. > > > > > > > > > > Wdyt Francisco ? > > > > > > > > > > PS: the downside of using globals is making things a bit slower as > u > > > need > > > > > to access extra storage for executing process instances. > > > > > > > > > > El mar, 7 may 2024, 15:21, Francisco Javier Tirado Sarti < > > > > > ftira...@redhat.com> escribió: > > > > > > > > > > > Alex, > > > > > > I think, in general, you use -1 too generously (the same applies > to > > > > > change > > > > > > requested status in PRs) > > > > > > First of all, this is using [DISCUSS] etiquette, so there is not > > > > really a > > > > > > proposal to -1 > > > > > > Second, in my latest e-mail, based on your feedback, I summarize > the > > > > > > options (context or not context), so I think the proper answer > to my > > > > > e-mail > > > > > > should be "I prefer option 2, not context, master workflow", > which is > > > > > what > > > > > > I deduce from your -1. > > > > > > Hopefully, I will be able to convince the user to use the second > > > > > approach. > > > > > > > > > > > > > > > > > > On Tue, May 7, 2024 at 2:47 PM Alex Porcelli <a...@porcelli.me> > > > wrote: > > > > > > > > > > > > > I understand the willingness to simplify, but I think this > risks > > > > > > > opening a can of worms. As mentioned by Enrique, Workflow is > not a > > > > > > > system of records... events are the way to go. > > > > > > > > > > > > > > Apache KIE is a code-first approach, so I'd argue that it's > > > expected > > > > > > > that users will need to do some coding, and leverage best > practices > > > > to > > > > > > > best decouple workflows into smaller event based flows that > Apache > > > > KIE > > > > > > > already support out of the box. > > > > > > > > > > > > > > -1 for me. > > > > > > > > > > > > > > On Tue, May 7, 2024 at 5:53 AM Francisco Javier Tirado Sarti > > > > > > > <ftira...@redhat.com> wrote: > > > > > > > > > > > > > > > > Hi Enrique, > > > > > > > > Thanks for the explanation > > > > > > > > As I said, the proposal is trying to implement the concept of > > > > shared > > > > > > > > context information between workflows. > > > > > > > > In a particular domain, several and *independent* workflows > might > > > > > want > > > > > > to > > > > > > > > share the same set of process variables (and its values). > > > > > > > > One way of implementing that, without really changing the > core > > > you > > > > > are > > > > > > > > mentioning, is to have a workflow that sets the context (the > > > > context > > > > > > > > setting can be automated itself) and the other workflows (the > > > > domain > > > > > > > ones) > > > > > > > > that use it by including the context as part of the input > > > > parameters. > > > > > > > > The other way will be for workflows (of that particular > domain) > > > to > > > > > > > > explicitly read the context using some primitive (so you will > > > have > > > > to > > > > > > > know > > > > > > > > that a particular variable is in the "context"). I found this > > > > latter > > > > > > less > > > > > > > > desirable than the former. The workflow designer knows that > the > > > > > > workflow > > > > > > > > has a certain input, it really does not care how this input > is > > > > > > provided. > > > > > > > > It is true that through events or REST invocation, this > merging > > > can > > > > > be > > > > > > > > manually done (in fact, this is what the user is currently > doing, > > > > it > > > > > > > reads > > > > > > > > the output from Data index and manually add to the input of > the > > > > > > workflow > > > > > > > to > > > > > > > > be executed), but I think it is a reasonable request to ask > for > > > > some > > > > > > > > facilities to perform this merge using just runtimes > > > infrastructure > > > > > > > > (without DI being present). > > > > > > > > It is true that the whole domain, at least in theory, can be > > > > modelled > > > > > > as > > > > > > > a > > > > > > > > huge workflow (this was in fact my first proposal to the > user, > > > but > > > > > they > > > > > > > > were not really appealed by it, I can try again though), > where > > > the > > > > > > first > > > > > > > > part is the "context" setting and then you can have > multiple > > > > > > branches > > > > > > > > (one per each independent workflow) waiting for human > > > intervention. > > > > > In > > > > > > > this > > > > > > > > approach, every independent workflow can be invoked as a > subflow > > > > > > > (triggered > > > > > > > > by an event or http call). But we need to check how the whole > > > thing > > > > > > will > > > > > > > be > > > > > > > > visualized in DataIndex. > > > > > > > > So far, there are following options > > > > > > > > 1) Implement shared context through input parameters. There > are > > > two > > > > > > > > alternative implementations: > > > > > > > > a) Change Kogito ProcessService (which is not part of the > engine) > > > > to > > > > > > > > perform the merge if configuration is present and add > interface > > > to > > > > > > > > existence persistence addon to store the output when the > process > > > > > > finishes > > > > > > > > if the configuration is present, > > > > > > > > b) Do not touch any existing class, add an addon (only > active for > > > > > users > > > > > > > > that want this behaviour) that listen for process start and > > > perform > > > > > the > > > > > > > > merge there, with an additional addon for each supported DB > to > > > read > > > > > the > > > > > > > > data to be merged-, This is more implementation work than > a), so > > > > you > > > > > > now > > > > > > > > understand why I'm writing so much to adopt a) :) > > > > > > > > 2) Do not implement shared context, but write the whole > domain > > > as a > > > > > > > single > > > > > > > > workflow that invokes independent workflows as subflows > based on > > > > > event > > > > > > > > triggering. We need to check how this is visualized in Data > Index > > > > and > > > > > > > > monitoring. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 7:11 PM Enrique Gonzalez Martinez < > > > > > > > > egonza...@apache.org> wrote: > > > > > > > > > > > > > > > > > Hi Francisco, > > > > > > > > > Workflow engine is based at the very core in two different > > > > > > primitives: > > > > > > > > > events and activities. > > > > > > > > > > > > > > > > > > Events are something that happens that initiates and > execution. > > > > > > > > > Catch/throw events of any type, timers, conditions... > > > > > > > > > An activity is something that produces some sort of > computation > > > > > unit > > > > > > > like > > > > > > > > > script, user taks, etc.. > > > > > > > > > > > > > > > > > > Anything that starts a process should be achieved by an > event > > > > > > (start), > > > > > > > and > > > > > > > > > a human task is not really an event of any kind. > > > > > > > > > > > > > > > > > > There is a primitive called start event with parallel > multiple > > > > set > > > > > to > > > > > > > true > > > > > > > > > that covers multiple events definitions that must happen in > > > order > > > > > to > > > > > > > start > > > > > > > > > a process. We don't have this sort of implementation right > now > > > in > > > > > the > > > > > > > > > engine.( It is really hard to make it work and requires > careful > > > > > > > design) but > > > > > > > > > should be your use case if we had it. One event for process > > > being > > > > > > > finished > > > > > > > > > and other event triggered by some human sending a start > with > > > its > > > > > > > inputs. > > > > > > > > > > > > > > > > > > That being said, about this idea: > > > > > > > > > > > > > > > > > > This sort of idea will introduce into the very core of the > > > engine > > > > > > human > > > > > > > > > tasks as events. I dont see any justification to do that. > > > > > > > > > > > > > > > > > > As we did mention at some point, human task will become an > > > entire > > > > > > > subsystem > > > > > > > > > on its own as it does not fit the requirements to be a > proper > > > > > > subsystem > > > > > > > > > with the features we had in v7. > > > > > > > > > > > > > > > > > > This introduce a concept of process orchestration based on > > > humans > > > > > > input > > > > > > > > > which defeats the purpose of a workflow as you are > introducing > > > an > > > > > > > arbitrary > > > > > > > > > way of subprocess execution or interdependent process > based on > > > > > > humans. > > > > > > > It > > > > > > > > > is not the same of using the output of a human task to > trigger > > > > the > > > > > > > > > execution of a subprocess that using human input as a some > sort > > > > of > > > > > > > gateway > > > > > > > > > event. > > > > > > > > > > > > > > > > > > How to perform this: > > > > > > > > > As Alex mention you can achieve this in a very simple way. > > > > > > > > > 1. Process finish and send a message to a kafka queue > > > > > > > > > 2 - third party system gets the events and allows you to > > > > manipulate > > > > > > the > > > > > > > > > input > > > > > > > > > 3 - it sends to a queue > > > > > > > > > 4 - process listening to kafka triggers a start event. > > > > > > > > > > > > > > > > > > If you dont like the third party system you can create a > very > > > > > simple > > > > > > > > > process that reads from a stream, allows you to modify the > > > input > > > > > and > > > > > > > sends > > > > > > > > > the outcome to another stream. > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > El lun, 6 may 2024, 17:16, Francisco Javier Tirado Sarti < > > > > > > > > > ftira...@redhat.com> escribió: > > > > > > > > > > > > > > > > > > > Alex, I might be missing something, but I do not think > this > > > > > > scenario > > > > > > > can > > > > > > > > > be > > > > > > > > > > covered through event consumption. The key part is that > > > > workflows > > > > > > of > > > > > > > > > type B > > > > > > > > > > are manually executed by users, which will provide its > own > > > set > > > > of > > > > > > > > > > parameters. Workflow of type A is just setting a variable > > > > context > > > > > > > which > > > > > > > > > is > > > > > > > > > > shared by all workflows of type B. To simulate such > context > > > > > without > > > > > > > > > > introducing the concept into the workflow definition > itself, > > > > the > > > > > > > > > properties > > > > > > > > > > setup by A should be passed as input of B. > > > > > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 5:05 PM Alex Porcelli < > > > > > porce...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Isn’t this already achievable using events with > different > > > > > topics? > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 11:02 AM Francisco Javier Tirado > > > > Sarti < > > > > > > > > > > > ftira...@redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > This is related with issue > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-kie-kogito-runtimes/issues/3495 > > > > > > > > > > > > We have one user which would like to reuse the > result of > > > > one > > > > > > > workflow > > > > > > > > > > > > execution (let's call this workflow of type A) as > input > > > of > > > > > > > several > > > > > > > > > > > > workflows (lets call them workflows of type B) > > > > > > > > > > > > > > > > > > > > > > > > Workflow A is executed before all B workflows. Then B > > > > > workflows > > > > > > > are > > > > > > > > > > > > manually executed by users. The desired input of B > > > > workflows > > > > > > > should > > > > > > > > > > be a > > > > > > > > > > > > merge of what the user provides when performing the > start > > > > > > > request and > > > > > > > > > > the > > > > > > > > > > > > output of workflow A. In order to achieve this, it is > > > > > expected > > > > > > > that > > > > > > > > > > users > > > > > > > > > > > > include, in the start request of workflow of type > B, the > > > > > > process > > > > > > > > > > > instance > > > > > > > > > > > > id of workflow A (so rather than taking the output > of A > > > and > > > > > > > merging > > > > > > > > > it > > > > > > > > > > > for > > > > > > > > > > > > every call, they just pass the process instance id) > > > > > > > > > > > > > > > > > > > > > > > > In order for this approach to work, output of > workflow A > > > > has > > > > > to > > > > > > > be > > > > > > > > > > stored > > > > > > > > > > > > somewhere in the DB (Currently runtimes DB only > stores > > > > active > > > > > > > process > > > > > > > > > > > > information). Since we do not want all process to > keep > > > > their > > > > > > > output > > > > > > > > > > > > information in the DB (only workflows of type A), > > > workflows > > > > > of > > > > > > > type A > > > > > > > > > > has > > > > > > > > > > > > to be identified somehow > > > > > > > > > > > > > > > > > > > > > > > > But before entering into more implementation details, > > > > which I > > > > > > > would > > > > > > > > > > like > > > > > > > > > > > to > > > > > > > > > > > > know is if this is a valid case both for BPMN or > not. The > > > > > > > > > > implementation > > > > > > > > > > > > implications are pretty relevant. If a valid use > case for > > > > > both > > > > > > > BPMN > > > > > > > > > and > > > > > > > > > > > > SWF, we can implement this functionality in the > kogito > > > > core, > > > > > > > there we > > > > > > > > > > can > > > > > > > > > > > > take advantage of existing persistence addons and > add the > > > > > newly > > > > > > > > > > required > > > > > > > > > > > > storage there. If not, we need to provide a SWF > specific > > > > > addon > > > > > > > for > > > > > > > > > each > > > > > > > > > > > > existing persistence add-on with the additional > storage. > > > > > > > > > > > > Let's share your thoughts. > > > > > > > > > > > > Thanks in advance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > For additional commands, e-mail: dev-h...@kie.apache.org > >
