Hi Enrique,
Firstly, apologies as I cannot see your architecture images.. they must be
getting striped… or perhaps I’m being daft..
Like the idea.. but.. a long running process has the real possibility of
already having computed or made decisions on some inputs (at the start of and
also intermediate to the process). I think this means that migration of this
process to a new version can only reliably occur if the proposed change only
kicks in after the current state of the process – otherwise you risk problems
with mis-matching schemas or message state.
I’m struggling to see how you can reliably handle this. As an example lets say
the process has several internal states (1 and 2 below) where it waits for
outside input from another client:
Process: Start -> 1-> 2 -> Done
Let’s say I sent message with schema ‘A’ into the process at Start and it
transforms it to a message with schema ‘B’ and sends it to another entity at
state ‘1’ which sends back a modification before it is sent on to state 2 and
eventually Done.
* If I modify the process to say add another step after 2, called 3 and
perform the migration just after this process reaches stage 2 then it should
work.. given all else is the same:
* Start -> 1-> 2 -> 3-> Done
* But what if I modify the process by introducing a step 1.5 and try to
migrate? In this case the message at state 2 has completely missed the
processing at 1.5 and is potentially in-consistent with the process. Its
schema could even have been changed (though if that then we’ve likely Also
changed handlers at 2 and beyond.
* Start -> 1-> 1.5 -> 2 -> Done
It seems to me that the only way to manage this kind of thing is indeed to
publish another version of the process to handle any new messages and allow the
original one to complete in its own time. Or to enable it to be stopped. Note
that step 2 could have sent a message to some company internal application with
makes a decision and changes something due to the process. How do we un-pick
that change?
From: Enrique Gonzalez Martinez <[email protected]>
Sent: Friday, February 9, 2024 7:25 AM
To: [email protected]
Subject: [PROPOSAL] Process Instance Migration Proposal
Process Instance Migration Proposal
Long running process may have an extended lifecycle that, in such a period,
inevitably process definition tweaks will be required for reasons that might be
related to business needs or even bug fixes. For that it's important to offer
users a mechanism to move their active running instances from one version to
the next
The routing problem will be achieved by the transport tier and signal manager
and it is not within the scope of this proposal. (How to signal a specific
container which contains the process instance or how the signal reaches its
destination within the container).
This document describes the process called migration for moving one process
version to another in a native cloud environment.
Constraints
This section enumerates which are the restrictions we have:
1. Deployments: We do not have servers anymore. We are working with
immutable images. We need to get rid of the idea of deployment
2. Avoid use of multiple versions in the same image. Having the
definitions of the old processes should not be needed anymore.
3. Avoid in memory processing migration. This ends up being very slow and
really hard to optimize.
Architecture
The solution proposed relies on 3 different base concepts:
· Process Instance Migration Plan File (MP): a file containing source and
target process definitions and its versions plus a node mapping between sources
and targets.
· Process Instance Migration Subsystem (PIM): component that takes a MP
and process state as inputs migrate to outputs. As any other subsystem will
throw an event.
· Process Instance Marshalling Decorator: An instrumentation approach
happening during unmarshalling of the process that can modify the process
instance state.
The dependency structure will look like this:
[cid:ii_lsebm2my0]
The interaction will work like this:
[cid:ii_lsebm2n71]
This leads to two things:
1. Get rid of in memory bulk process instance migration. It will be done
effectively during next operation and eventually being consistent. (we do it
during unmarshalling using a process marshaller extension point)
2. Get rid of all the process definitions involved in the migration in
the same container (we have the migration plan)
There are some requirements to achieve this.
1. We need to supply a migration plan with information known by the user,
otherwise it won’t be possible to build the file.
2. Process marshaller decorator should be really fast. (We need to remove
any computational work during unmarshalling) (just replacing UUID should be
enough.
Code assessment
This section describes what are the problems found in the current based code
does not allow to make this happen.
If we revisit the current Migration Manager in v7 the main problem is that the
system requires some computation because it needs to convert the internal
identifiers of any workflow element and then replace to the new ones. This
internal representation enforces two things:
· the deployment of the old process definition as a container in v7, as
we need to match the id and then move one to the other. The internal
representation depends on an algorithm of how workflow is being parsed and
processed.
· Requires the computation step to make lookups through the process
definitions and compute them to the new ones.
Removing the internal representation of the node and node instance will cause:
· No computation required: There won’t be any need for this step. The
computation step would be replaced entirely by the migration plan (as it will
be self-contained and not split by migration plan and internal representation).
· No old process definition required. This does not require anymore as it
is being replaced as well by the migration plan and algorithm.
