opened issue for message schema multiversion https://github.com/apache/incubator-kie-issues/issues/924
El lun, 12 feb 2024 a las 19:25, Enrique Gonzalez Martinez (< [email protected]>) escribió: > I found really interesting the problem you are pointing out. We were > focusing the traditional problem of migrating the process state but not > really what happens with the payload sent/received by external systems or > even in our case even other engines. > > They are different problems because that not really affects the state but > the message processing in the engine. > > I will open a task to investigate further, but i guess the only way to > support inflight processing of older messages is to provide some sort of > converter at endpoint level or even using kafka streams processor to do > such transformation for the engine. This seems fixable either by some > custom code in the engine or stream processors. > > Have you give a thought about this ? Do you have a case to share ? > > El lun, 12 feb 2024, 18:20, Jon Ferguson <[email protected]> > escribió: > >> Cheers... >> >> I link I get the approach - looks like you are essentially enabling the >> process instance to 'change itself' by following the map rather than doing >> surgery on the process instance from the outside. Nice idea. >> >> Unfortunately, I don't know enough about the code yet nor how effective >> the old Migration Manager was in managing in-process migrations to see how >> this would work with real processes. But your approach does look like it >> could reduce complexity significantly. >> >> " The engine is not responsible for the semantics or compatibility of >> your process definition." I think you're saying here that the change >> introduced by a new process version/bug fix has to be compatible with the >> old process. I can introduce a new step 1.5 but it better not result in, >> incompatible message transformations otherwise process instances that are >> further along in processing (say they are currently at step 3 when the >> process instance is migrated) may be working with incompatible message >> types (original step 3 expected input schema S, new step 3 expects schema >> S' due to transformations introduced in step 1.5). From your comments, >> this is simply out of scope for this proposal. That's ok. On the other >> hand I can see that the migration plan could also introduce any required >> transformations that might be required to address this. >> >> I find this quite interesting because of an application where a new >> incoming message schema may be imposed on a system running workflows that >> drive transformations and machine driven interactions that may be >> long-running. Assuming you have N different systems interacting through >> this one BPMN process, and each of these would need to be upgraded to cope >> with the new schema but which are independently deployed, how then do you >> do a safe migration? From a message schema p.o.v. you must need to be able >> to transform both ways between the versions of the schema and intelligently >> apply this to the endpoints. Avro has a simple mechanism that allows >> *some* compatible changes, and I think processes must have a similar >> limit. Eg. You can't really do much before things start to break. Maybe >> just introduce a new manual sign-off step! It would just be very cool to >> be able to define a process whereby all N systems could independently >> upgrade to the new schema/process without breaking anything and without >> stopping these long-running processes. >> >> -----Original Message----- >> From: Enrique Gonzalez Martinez <[email protected]> >> Sent: Monday, February 12, 2024 3:00 PM >> To: [email protected] >> Subject: Re: [PROPOSAL] Process Instance Migration Proposal >> >> you can get more info in here >> >> https://github.com/apache/incubator-kie-issues/issues/861 >> >> The original document is in there. >> >> Answering inline >> >> El lun, 12 feb 2024 a las 15:47, Jon Ferguson (<[email protected]>) >> escribió: >> >> > Hi Enrique, >> > >> > Firstly, apologies as I cannot see your architecture images.. they >> > must be getting striped… or perhaps I’m being daft.. >> > >> > Like the idea.. but.. a long running process has the real possibility >> > of already having computed or made decisions on some inputs (at the >> > start of and also intermediate to the process). I think this means >> > that migration of this process to a new version can only reliably >> > occur if the proposed change only kicks in after the current state of >> > the process – otherwise you risk problems with mis-matching schemas or >> message state. >> > >> > >> Once the process instance is marked it is just a matter of translating >> old nodes to the new ones. >> >> >> > I’m struggling to see how you can reliably handle this. As an example >> > lets say the process has several internal states (1 and 2 below) >> > where it waits for outside input from another client: >> > >> > Process: Start -> 1-> 2 -> Done >> > >> > Let’s say I sent message with schema ‘A’ into the process at Start >> > and it transforms it to a message with schema ‘B’ and sends it to >> > another entity at state ‘1’ which sends back a modification before it >> > is sent on to state >> > 2 and eventually Done. >> > >> > * If I modify the process to say add another step after 2, called 3 >> > and perform the migration just after this process reaches stage 2 then >> > it should work.. given all else is the same: >> > * Start -> 1-> 2 -> 3-> Done >> > >> >> >> > * But what if I modify the process by introducing a step 1.5 and try >> > to migrate? In this case the message at state 2 has completely missed >> > the processing at 1.5 and is potentially in-consistent with the >> > process. Its schema could even have been changed (though if that then >> > we’ve likely Also changed handlers at 2 and beyond. >> > * Start -> 1-> 1.5 -> 2 -> Done >> > >> >> The migration plan should contain how to translate those nodes changed or >> deleted in the new process definition. >> In this case you have almost all cases: >> 1 it is an idle state (catch event, human task... some of the sort)... >> then when you move to then next version it won't have to do anything as the >> map is 1 to 1 let's say it is 2 and the equivalent in the new process >> instance is 1.5 then you need to set your migration plan from 2 to 1.5 >> Let's say 2 stays the same... it means your process will remain 2 and will >> still be consistent (if you have not changed that node the user will be >> responsible for making it compatible) Let's say you are in 3... then you >> will need to translate that into a new state. >> >> The engine is not responsible for the semantics or compatibility of your >> process definition... It was never meant that way. At least in jbpm. We >> might think of some improvements (any suggestion is accepted) but the main >> goal right now is to have the tool to have process instances migrated >> between versions without a bulk in memory process. >> >> The migration plan can hold anything we want (rules included) but that is >> something we need to bake in upstream carefully. as long as the approach is >> correct we can grow anything from there. >> >> >> >> >> > >> > It seems to me that the only way to manage this kind of thing is >> > indeed to publish another version of the process to handle any new >> > messages and allow the original one to complete in its own time. Or >> > to enable it to be stopped. Note that step 2 could have sent a >> > message to some company internal application with makes a decision and >> > changes something due to the process. How do we un-pick that change? >> > >> > >> > >> > >> > From: Enrique Gonzalez Martinez <[email protected]> >> > Sent: Friday, February 9, 2024 7:25 AM >> > To: [email protected] >> > Subject: [PROPOSAL] Process Instance Migration Proposal >> > >> > >> > Process Instance Migration Proposal >> > >> > Long running process may have an extended lifecycle that, in such a >> > period, inevitably process definition tweaks will be required for >> > reasons that might be related to business needs or even bug fixes. For >> > that it's important to offer users a mechanism to move their active >> > running instances from one version to the next >> > >> > The routing problem will be achieved by the transport tier and signal >> > manager and it is not within the scope of this proposal. (How to >> > signal a specific container which contains the process instance or how >> > the signal reaches its destination within the container). >> > >> > This document describes the process called migration for moving one >> > process version to another in a native cloud environment. >> > >> > Constraints >> > >> > This section enumerates which are the restrictions we have: >> > >> > 1. Deployments: We do not have servers anymore. We are working >> with >> > immutable images. We need to get rid of the idea of deployment >> > >> > 2. Avoid use of multiple versions in the same image. Having the >> > definitions of the old processes should not be needed anymore. >> > >> > 3. Avoid in memory processing migration. This ends up being very >> > slow and really hard to optimize. >> > >> > Architecture >> > >> > The solution proposed relies on 3 different base concepts: >> > >> > · Process Instance Migration Plan File (MP): a file containing >> > source and target process definitions and its versions plus a node >> > mapping between sources and targets. >> > >> > · Process Instance Migration Subsystem (PIM): component that >> takes a >> > MP and process state as inputs migrate to outputs. As any other >> > subsystem will throw an event. >> > >> > · Process Instance Marshalling Decorator: An instrumentation >> > approach happening during unmarshalling of the process that can modify >> > the process instance state. >> > >> > The dependency structure will look like this: >> > >> > [cid:ii_lsebm2my0] >> > >> > The interaction will work like this: >> > >> > [cid:ii_lsebm2n71] >> > >> > >> > >> > >> > >> > This leads to two things: >> > >> > 1. Get rid of in memory bulk process instance migration. It will >> be >> > done effectively during next operation and eventually being >> > consistent. (we do it during unmarshalling using a process marshaller >> > extension point) >> > >> > 2. Get rid of all the process definitions involved in the >> migration >> > in the same container (we have the migration plan) >> > >> > There are some requirements to achieve this. >> > >> > 1. We need to supply a migration plan with information known by >> the >> > user, otherwise it won’t be possible to build the file. >> > >> > 2. Process marshaller decorator should be really fast. (We need to >> > remove any computational work during unmarshalling) (just replacing >> > UUID should be enough. >> > >> > Code assessment >> > >> > This section describes what are the problems found in the current >> > based code does not allow to make this happen. >> > >> > If we revisit the current Migration Manager in v7 the main problem is >> > that the system requires some computation because it needs to convert >> > the internal identifiers of any workflow element and then replace to >> > the new ones. This internal representation enforces two things: >> > >> > · the deployment of the old process definition as a container in >> v7, >> > as we need to match the id and then move one to the other. The >> > internal representation depends on an algorithm of how workflow is >> > being parsed and processed. >> > >> > · Requires the computation step to make lookups through the >> process >> > definitions and compute them to the new ones. >> > >> > Removing the internal representation of the node and node instance >> > will >> > cause: >> > >> > · No computation required: There won’t be any need for this step. >> > The computation step would be replaced entirely by the migration plan >> > (as it will be self-contained and not split by migration plan and >> > internal representation). >> > >> > · No old process definition required. This does not require >> anymore >> > as it is being replaced as well by the migration plan and algorithm. >> > >> > >> > >> >> -- >> Saludos, Enrique González Martínez :) >> >
