1. Let the platform to do the work giving the information to the platform required and let use user to query that info. This is our preferred way. We just need to build a way to provide that information and the platform has its own tools to query metadata from the deployment. We just need a common ground to define the info structure.
2. Write our own tool capturing events from the deployments whenever deploys or not. This requires more effort as you will need to clutter the data index and mix responsibilities in there The option that from my point of view is the first one. The advantages are clear imo 1. Platform requires a different skill set and tools platform dependent. Moving this to our system will mean to try to solve a buch of stuff in an independent way making things really hard and making devops needed to learn new stuff and not being able to use their tooling. That is and additional operation cost 2. It is easier to provide a driver to set metadata in the deployment than creating custom drivers to solve properly events in a platform constantly changing 3. Service url is a fragile approach in a cloud environment. It is too error prone and couple microservices to the deployment environment El jue, 18 jul 2024, 10:18, Enrique Gonzalez Martinez <[email protected]> escribió: > Hi Francisco, > > Just to clarify. We are not against having some sort of way to tell which > process definitions are deployed or not. But you need to keep in mind that > there are two different ways to do this: > > El jue, 18 jul 2024, 10:01, Francisco Javier Tirado Sarti < > [email protected]> escribió: > >> Yes, I understand that your preference is to delegate deployment status to >> an external system. >> However I would like to explore the consequences of that approach and how >> things are supposed to work. The first thing that come to mind is that the >> user of the platform will be forced to have two different graphical tools >> for tracking: one for process definition deployment (the one that will >> include the deployment status and will communicate with the user custom >> db) and the one provided by the platform for process definition instances >> (current runtime tooling communicating with DI) >> Are we sure we are fine with this architecture? >> >> On Wed, Jul 17, 2024 at 7:48 PM Alex Porcelli <[email protected]> wrote: >> >> > Francisco, >> > >> > Seems that a few of us believe this would be better addressed by an >> > external system. >> > >> > >> > On Wed, Jul 17, 2024 at 7:14 AM Francisco Javier Tirado Sarti < >> > [email protected]> wrote: >> > >> > > Hi, >> > > Yes, process definition is used because runtime tooling is relying on >> > > process definition to be able to render a graphical representation of >> the >> > > runtime progress. >> > > It can be argued that a runtime tool might want to know in which >> servers >> > of >> > > the cluster that process definition is deployed and the status of that >> > > deployment. So, the same serviceURL is there, a deployment status >> flag is >> > > not a crazy idea. >> > > >> > > On Thu, Jul 11, 2024 at 6:45 PM Alex Porcelli <[email protected]> >> wrote: >> > > >> > > > Francisco, >> > > > >> > > > You're spot on on the purpose of the current Process Definition on >> DI: >> > > > it provides complementary information to process instances. >> > > > >> > > > Example? Process definition (ie. diagram) for the current instance. >> > > > >> > > > On Thu, Jul 11, 2024 at 12:48 PM Francisco Javier Tirado Sarti >> > > > <[email protected]> wrote: >> > > > > >> > > > > Hi, >> > > > > Since DI already has a ProcessDefinition entity with a service >> URL, >> > we >> > > > feel >> > > > > that adding deployment status was not breaking anything that was >> not >> > > > > already broken. >> > > > > I guess it all depends on what we think should be the purpose of >> DI. >> > > > > If it is just to offer information about process instances, >> Enrique >> > is >> > > > > right, this is not the place to monitor which process definitions >> are >> > > > > available or not. >> > > > > However, if the purpose of DI is just that, why is the process >> > > definition >> > > > > entity there? as supporting information for the process instance >> > only? >> > > > > I feel we need to clearly define this before discarding Walters' >> > > > proposal. >> > > > > If we take a strict approach, then it means that any software >> > deploying >> > > > > runtimes and data index will have to develop its own deployment >> > > component >> > > > > with its own db. That's probably fair, but should be written >> > somewhere. >> > > > > >> > > > > >> > > > > >> > > > > On Tue, Jul 9, 2024 at 3:39 PM Alex Porcelli <[email protected]> >> > wrote: >> > > > > >> > > > > > Walter, >> > > > > > >> > > > > > I have to agree with Enrique… although your ask is minimal in >> the >> > > code >> > > > > > changes and impact, the problem is more on the message that it >> > sends. >> > > > > > >> > > > > > As pointed out by last email from Enrique, we - as Apache KIE - >> > don’t >> > > > > > control or have visibility over the whole environment, so it’s >> > > > virtually >> > > > > > impossible to guarantee that something is set to not available >> is >> > > > really >> > > > > > not available, and vice versa. >> > > > > > >> > > > > > I understand that the serverless operator aims to provide an >> > > > experience, >> > > > > > but it also has limitation over the installation… so probably >> this >> > > > kind of >> > > > > > information would be better stored outside the Apache KIE >> systems / >> > > in >> > > > your >> > > > > > case, in the downstream product database. >> > > > > > >> > > > > > Regards, >> > > > > > _____________ >> > > > > > Alex Porcelli >> > > > > > http://porcelli.me >> > > > > > >> > > > > > >> > > > > > On Tue, Jul 9, 2024 at 1:59 AM Enrique Gonzalez Martinez < >> > > > > > [email protected]> wrote: >> > > > > > >> > > > > > > Hi Walter, >> > > > > > > >> > > > > > > I see some problems aside from going on in mixing the data >> index >> > > with >> > > > > > data >> > > > > > > does not belong to it like deployment mgmt. E.g. The service >> url >> > is >> > > > there >> > > > > > > not because the DI but the gateway component (mutations) that >> > makes >> > > > > > changes >> > > > > > > in runtimes. Something that needs to change in the future. >> But i >> > > > digress. >> > > > > > > >> > > > > > > Back to this is deployment management. Basically the >> requirement >> > is >> > > > to >> > > > > > > compute the current catalog of process definitions deployed in >> > the >> > > > > > > environment. >> > > > > > > >> > > > > > > I am not against but I am not comfortable mixing with the data >> > > index. >> > > > > > > Because it is not part of the data index goal; provide the >> last >> > > state >> > > > > > > information of the process instances. >> > > > > > > >> > > > > > > Regarding how this approach works and given you are capable of >> > > > counting >> > > > > > > properly instances. >> > > > > > > >> > > > > > > 1. Let's say you bring down one workflow. The system will be >> > marked >> > > > as >> > > > > > > undeployed/deleted. You don't specify how alive process >> instance >> > > > will be >> > > > > > > processed or if you will perform some other action. >> > > > > > > The other problem i see is trying to already solve platform >> > > > problems. I >> > > > > > > mean, if the platform already can compute which containers are >> > > > deployed >> > > > > > and >> > > > > > > therefore which is the proper catalog why should we polute the >> > > system >> > > > > > with >> > > > > > > this ? Would not make more sense in telling the user how to >> query >> > > > that >> > > > > > info >> > > > > > > in the platform ? Making platform independent does not give a >> > > proper >> > > > > > > benefits of chasing multiple platform problems regarding >> > > deployment / >> > > > > > > undeployment operations, url changes, counting instances... >> Will >> > > > never >> > > > > > get >> > > > > > > it right. It is an unsustainable approach. >> > > > > > > >> > > > > > > El lun, 8 jul 2024, 11:18, Walter Medvedeo < >> [email protected] >> > > >> > > > > > > escribió: >> > > > > > > >> > > > > > > > Hi Enrique, the requirement we have from SonataFlow users is >> > very >> > > > > > simple. >> > > > > > > > >> > > > > > > > Imagine the ProcesssDefinitions in the DI as a catalog of >> all >> > the >> > > > > > > existing >> > > > > > > > definitions. >> > > > > > > > >> > > > > > > > Similar to the sonataflow-console, the jbpm-console, etc, >> user >> > > > > > > > applications query the ProcessDefintions to get the list >> with >> > > all >> > > > the >> > > > > > > > existing definitions, and present that list for example in a >> > UI. >> > > > > > > > >> > > > > > > > So, imagine that with the ProcessDefinitions query you get >> the >> > > > > > following >> > > > > > > > list: >> > > > > > > > >> > > > > > > > purchase-order, serviceUrl_1 >> > > > > > > > new-customer, serviceUrl_2 >> > > > > > > > etc. >> > > > > > > > >> > > > > > > > Similar to what we do in our consoles, in their UIs, users >> can >> > > > pick any >> > > > > > > > process from that list to create new instances, etc. >> > > > > > > > >> > > > > > > > All good. >> > > > > > > > >> > > > > > > > In a later point in time, an administrator, decides that the >> > > > > > > > "new-customer" process is out from the catalog. We won't >> work >> > > with >> > > > that >> > > > > > > > ProcessDefinition anymore. >> > > > > > > > >> > > > > > > > Very simplified, In SonataFlow Operator driven setups, you >> can >> > do >> > > > it by >> > > > > > > > telling the operator to "un-deploy" the worfklow. As part >> of >> > > this >> > > > > > > > procedure, the operator can send a basic event to the data >> > index >> > > > > > telling >> > > > > > > > that situation. >> > > > > > > > >> > > > > > > > So, at the data index, we need to mark the "new-customer" >> > process >> > > > > > > > definition as "deleted", or better said as "logically >> deleted". >> > > > > > (Because, >> > > > > > > > as I explained in previous comments, we don't want, at >> least by >> > > > now, do >> > > > > > > > physical deletions in the DI) >> > > > > > > > >> > > > > > > > To do that, we need to add a "new field". (Simplest >> approach, >> > we >> > > > don't >> > > > > > > > want more elaborated modelings) >> > > > > > > > >> > > > > > > > I suggested to name that field as "status", to keep it >> > completely >> > > > > > > > unrelated from the type of setup/deployment, because, as I >> > > > mentioned, >> > > > > > > > kogito supports a plenty of setups, docker, docker compose, >> > > > Sonataflow >> > > > > > > > operator driven, standalone JVM, etc. >> > > > > > > > >> > > > > > > > So, introduced that we could for example mark "status" = >> > > > "unavailable", >> > > > > > > > and maybe this introduced confusion. If that is the case, >> > forget >> > > > about >> > > > > > > the >> > > > > > > > world unavailable, and to follow the reasoning, imagine we >> mark >> > > > that >> > > > > > > field >> > > > > > > > with "status" = "deleted" (logical deletion) >> > > > > > > > >> > > > > > > > Finally, by adding this simple field/concept, totally >> decoupled >> > > > from >> > > > > > the >> > > > > > > > various setups that people can use, we can facilitate >> > SonataFlow >> > > > users >> > > > > > > > applications to query the DI, and only get the >> > ProcessDefinitons >> > > > that >> > > > > > are >> > > > > > > > not "deleted" (logically deleted) >> > > > > > > > >> > > > > > > > Select * from ProcessDefinitions where status != deleted >> > > > > > > > >> > > > > > > > >> > > > > > > > Regards, >> > > > > > > > Walter. >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On 2024/07/04 11:35:08 Enrique Gonzalez Martinez wrote: >> > > > > > > > > Hi walter, >> > > > > > > > > I am not certain about the req. You have few things in >> motion >> > > in >> > > > here >> > > > > > > > > >> > > > > > > > > 1. The req says the deployment is still required : Then >> you >> > > have >> > > > to >> > > > > > do >> > > > > > > > > nothing as this information is just available in the >> index. >> > > > Query the >> > > > > > > > > active process instance of a certain procces id and >> version. >> > > You >> > > > > > dont >> > > > > > > > need >> > > > > > > > > to access the process definition. >> > > > > > > > > >> > > > > > > > > The req says you need to know if the deployment is >> required >> > but >> > > > you >> > > > > > did >> > > > > > > > not >> > > > > > > > > deploy data index. Then you need to check if there pricess >> > > > instance >> > > > > > > > > belonging to that deployment. >> > > > > > > > > >> > > > > > > > > The req says you need to know if a deployment is >> > > > > > unavailable/available >> > > > > > > i >> > > > > > > > > would say you need to send events related to that >> deployment >> > > and >> > > > this >> > > > > > > is >> > > > > > > > > more tricky. Because you need to somehow define first >> what is >> > > > > > > available. >> > > > > > > > It >> > > > > > > > > seems your definition is related to what is deployed. In >> that >> > > > sense i >> > > > > > > > would >> > > > > > > > > rely into the platform. As if you dont want to operate >> with >> > > > certain >> > > > > > > > process >> > > > > > > > > you just need to drop the container running it. It will be >> > > shown >> > > > in >> > > > > > the >> > > > > > > > > platform tool like dooker etc.... >> > > > > > > > > But if we want to keep track of history i will leave >> things >> > > > opening. >> > > > > > > For >> > > > > > > > > instance i am interested in we start issuing these events >> to >> > > > those >> > > > > > > > > deployments that are still managing some process instances >> > but >> > > > dont >> > > > > > > > accept >> > > > > > > > > new ones or deployments that have migrated the new process >> > > > > > definitions >> > > > > > > to >> > > > > > > > > new deployments...so i would try to not mess too much with >> > the >> > > > > > process >> > > > > > > > > definition but afd a new event in this case open to new >> > > > transition >> > > > > > and >> > > > > > > > > states.... And you can store that in any way you fancy in >> the >> > > > data >> > > > > > > index. >> > > > > > > > > >> > > > > > > > > El jue, 4 jul 2024, 13:03, Walter Medvedeo < >> > > [email protected] >> > > > > >> > > > > > > > escribió: >> > > > > > > > > >> > > > > > > > > > Hi Enrique, >> > > > > > > > > > maybe I couldn't express the concept well. >> > > > > > > > > > >> > > > > > > > > > But let's say that the DI has the catalog of all the >> > existing >> > > > > > > > > > ProcessDefinitions. >> > > > > > > > > > >> > > > > > > > > > To mark a definition as "unavailable" is kind of >> logically >> > > > remove >> > > > > > it >> > > > > > > > from >> > > > > > > > > > the Catalog. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On 2024/07/04 09:59:00 Enrique Gonzalez Martinez wrote: >> > > > > > > > > > > Hi walter, this won't give you the data you are aiming >> > for. >> > > > > > > > > > > >> > > > > > > > > > > If you have two instances and one goes down the state >> > will >> > > > be set >> > > > > > > as >> > > > > > > > not >> > > > > > > > > > > available unless you compute something in the data >> index >> > so >> > > > at >> > > > > > > least >> > > > > > > > you >> > > > > > > > > > > will need more information or duplicate lines. This is >> > > > unlikely >> > > > > > to >> > > > > > > > be the >> > > > > > > > > > > responsibility of the data index. >> > > > > > > > > > > >> > > > > > > > > > > El jue, 4 jul 2024, 11:28, Walter Medvedeo < >> > > > [email protected] >> > > > > > > >> > > > > > > > > > escribió: >> > > > > > > > > > > >> > > > > > > > > > > > Hi, the idea is to introduce the requirement/concept >> > > > first, and >> > > > > > > > > > summarize >> > > > > > > > > > > > the impact on the information registered in the DI >> as >> > > much >> > > > as >> > > > > > > > possible. >> > > > > > > > > > > > >> > > > > > > > > > > > And thus, very summarized, the proposal is to >> include >> > one >> > > > more >> > > > > > > > filed in >> > > > > > > > > > > > the ProcessDefinitions information registered by the >> > DI, >> > > > that >> > > > > > > will >> > > > > > > > let >> > > > > > > > > > us >> > > > > > > > > > > > know if a given definition is still "available" to >> > create >> > > > > > > > instances. >> > > > > > > > > > > > >> > > > > > > > > > > > That is a requirement we have from SonataFlow users. >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > On 2024/07/03 20:50:20 Jason Porter wrote: >> > > > > > > > > > > > > I am not exactly sure what the proposal is here. I >> > see >> > > a >> > > > > > bunch >> > > > > > > of >> > > > > > > > > > > > background info for this new feature, but I don't >> see >> > an >> > > > actual >> > > > > > > > > > proposal. >> > > > > > > > > > > > > >> > > > > > > > > > > > > On 2024/07/03 15:06:21 Walter Medvedeo wrote: >> > > > > > > > > > > > > > Hello Guys, >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Let me share some requirement we have. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Right now, as part of the ProcessDefinition >> > > > information we >> > > > > > > > store >> > > > > > > > > > in the >> > > > > > > > > > > > > > Data Index, we record for example the processId >> > and >> > > > the >> > > > > > > > > > serviceURL. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > That information is good for users to know, for >> > > > instance, >> > > > > > > which >> > > > > > > > > > are the >> > > > > > > > > > > > > > existing ProcessDefitnions, and, based on that >> > > > information, >> > > > > > > > they >> > > > > > > > > > can >> > > > > > > > > > > > for >> > > > > > > > > > > > > > example start a new process instance by >> > concatenating >> > > > the >> > > > > > > > > > serviceURL + >> > > > > > > > > > > > “/” >> > > > > > > > > > > > > > + processId. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > In the context of a SonataFlow Operator driven >> > > setups, >> > > > > > users >> > > > > > > > need >> > > > > > > > > > to >> > > > > > > > > > > > know >> > > > > > > > > > > > > > if a ProcessDefinition is still “available”. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > For example, users can deploy a WF, use it for >> some >> > > > time, >> > > > > > and >> > > > > > > > then, >> > > > > > > > > > > > decide >> > > > > > > > > > > > > > to un-deploy it. From this point, no more >> instances >> > > of >> > > > that >> > > > > > > > > > workflow >> > > > > > > > > > > > can be >> > > > > > > > > > > > > > created. The user has just decided to remove >> that >> > > > > > deployment >> > > > > > > > from >> > > > > > > > > > the >> > > > > > > > > > > > > > ecosystem. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > However, in the data-index, that definition is >> > still >> > > > > > > recorded. >> > > > > > > > And >> > > > > > > > > > > > users >> > > > > > > > > > > > > > want to keep it there, since they want to keep >> > > > consistent >> > > > > > > > > > information >> > > > > > > > > > > > about >> > > > > > > > > > > > > > the already executed instances etc. (The >> intention >> > is >> > > > to >> > > > > > not >> > > > > > > > > > introduce >> > > > > > > > > > > > any >> > > > > > > > > > > > > > information deleting procedure at the DI, etc.) >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On the other hand, users need a way to filter >> which >> > > > > > > > definitions are >> > > > > > > > > > > > still >> > > > > > > > > > > > > > “available” by using the standard graphql api. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > What we are evaluating is to include a “status” >> > > field >> > > > in >> > > > > > the >> > > > > > > > > > > > > > ProcessDefinitions, with the values >> > > > available/unavailable, >> > > > > > > and, >> > > > > > > > > > that >> > > > > > > > > > > > will >> > > > > > > > > > > > > > be updated as usual via ClouldEvents. In the >> case >> > of >> > > > > > > > SonataFlow, >> > > > > > > > > > those >> > > > > > > > > > > > > > events will be produced by the operator, which >> > > governs >> > > > the >> > > > > > > > > > > > > > deployment/undeployment, etc. Other setups might >> > just >> > > > > > ignore >> > > > > > > > that >> > > > > > > > > > > > field. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Note: we are looking for a very minimalist >> > approach >> > > > that >> > > > > > > lets >> > > > > > > > us >> > > > > > > > > > > > record >> > > > > > > > > > > > > > the situation, and facilitate SonataFlow users >> to >> > > query >> > > > > > that >> > > > > > > > > > situation >> > > > > > > > > > > > by >> > > > > > > > > > > > > > using the standard Graphql api. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > This is not any sort of more elaborated >> clustering >> > > > > > > > > > > > > > management/monitoring/etc. solution. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Independently of fine grained details about the >> > event >> > > > to >> > > > > > > > introduce, >> > > > > > > > > > > > which >> > > > > > > > > > > > > > carries more or less the information we are >> already >> > > > sending >> > > > > > > in >> > > > > > > > the >> > > > > > > > > > > > other DI >> > > > > > > > > > > > > > events. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > From the point of view of let’s say non >> SonataFlow >> > > > setups, >> > > > > > do >> > > > > > > > you >> > > > > > > > > > guys >> > > > > > > > > > > > see >> > > > > > > > > > > > > > any objections? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Walter. >> > > > > > > > > > > > > > ResponderReenviar >> > > > > > > > > > > > > > Añadir reacción >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > >> > > > >> --------------------------------------------------------------------- >> > > > > > > > > > > > > To unsubscribe, e-mail: >> > [email protected] >> > > > > > > > > > > > > For additional commands, e-mail: >> > > [email protected] >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > >> > > > >> --------------------------------------------------------------------- >> > > > > > > > > > > > To unsubscribe, e-mail: >> [email protected] >> > > > > > > > > > > > For additional commands, e-mail: >> > [email protected] >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > --------------------------------------------------------------------- >> > > > > > > > > > To unsubscribe, e-mail: [email protected] >> > > > > > > > > > For additional commands, e-mail: >> [email protected] >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > >> --------------------------------------------------------------------- >> > > > > > > > To unsubscribe, e-mail: [email protected] >> > > > > > > > For additional commands, e-mail: [email protected] >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > > >> --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: [email protected] >> > > > For additional commands, e-mail: [email protected] >> > > > >> > > > >> > > >> > >> >
