Ok, here is what you actually need to implement at the application level.
Lets say we have to call 2 services in the following order:
- Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => v1a,
k2 => v2a]
- Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => v2ab,
k3 => v3b]
The change
from [ k1 => v1, k2 => v2, k3 => v3 ]
to [ k1 => v1a, k2 => v2ab, k3 => v3b ]
must happen in a single transaction.
Optimistic protocol to solve this:
Each cache key must have a field `otx`, which is a unique orchestrator TX
identifier - it must be a parameter passed to all the services. If `otx` is
set to some value it means that it is an intermediate key and is visible
only inside of some transaction, for the finalized key `otx` must be null -
it means the key is committed and visible for everyone.
Each cache value must have a field `ver` which is a version of that value.
For both fields (`otx` and `ver`) the safest way is to use UUID.
Workflow is the following:
Orchestrator starts the distributed transaction with `otx` = x and passes
this parameter to all the services.
Service A:
- does some computations
- stores [k1x => v1a, k2x => v2a] with TTL = Za
where
Za - left time from max Orchestrator TX duration after Service A
end
k1x, k2x - new temporary keys with field `otx` = x
v2a has updated version `ver`
- returns a set of updated keys and all the old versions to the
orchestrator
or just stores it in some special atomic cache like
[x => (k1 -> ver1, k2 -> ver2)] TTL = Za
Service B:
- retrieves the updated value k2x => v2a because it knows `otx` = x
- does computations
- stores [k2x => v2ab, k3x => v3b] TTL = Zb
- updates the set of updated keys like [x => (k1 -> ver1, k2 -> ver2, k3
-> ver3)] TTL = Zb
Service Committer (may be embedded into Orchestrator):
- takes all the updated keys and versions for `otx` = x
[x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)]
- in a single transaction checks value versions for all the old values
and replaces them with calculated new ones
- does cleanup of temporary keys and values
- in case of version mismatch or TX timeout just rollbacks and signals
to Orchestrator to restart the job with new `otx`
PROFIT!!
This approach even allows you to run independent parts of the graph in
parallel (with TX transfer you will always run only one at a time). Also it
does not require inventing any special fault tolerance technics because
Ignite caches are already fault tolerant and all the intermediate results
are virtually invisible and stored with TTL, thus in case of any crash you
will not have inconsistent state or garbage.
Sergi
2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>:
> Okay, we are open for proposals on business task. I mean, we can make use
> of some other thing, not distributed transaction. Not transaction yet.
>
> ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <voze...@gridgain.com>:
>
> > IMO the use case makes sense. However, as Sergi already mentioned, the
> > problem is far more complex, than simply passing TX state over a wire.
> Most
> > probably a kind of coordinator will be required still to manage all kinds
> > of failures. This task should be started with clean design proposal
> > explaining how we handle all these concurrent events. And only then, when
> > we understand all implications, we should move to development stage.
> >
> > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV <
> > alkuznetsov...@gmail.com> wrote:
> >
> > > Right
> > >
> > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <sergi.vlady...@gmail.com
> >:
> > >
> > > > Good! Basically your orchestrator just takes some predefined graph of
> > > > distributed services to be invoked, calls them by some kind of RPC
> and
> > > > passes the needed parameters between them, right?
> > > >
> > > > Sergi
> > > >
> > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com
> > >:
> > > >
> > > > > orchestrator is a custom thing. He is responsible for managing
> > business
> > > > > scenarios flows. Many nodes are involved in scenarios. They
> exchange
> > > data
> > > > > and folow one another. If you acquinted with BPMN framework, so
> > > > > orchestrator is like bpmn engine.
> > > > >
> > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <sergi.vlady...@gmail.com
> >:
> > > > >
> > > > > > What is Orchestrator for you? Is it a thing from Microsoft or
> your
> > > > custom
> > > > > > in-house software?
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <
> > > alkuznetsov...@gmail.com
> > > > >:
> > > > > >
> > > > > > > Fine. Let's say we've got multiple servers which fulfills
> custom
> > > > logic.
> > > > > > > This servers compound oriented graph (BPMN process) which
> > > controlled
> > > > by
> > > > > > > Orchestrator.
> > > > > > > For instance, *server1 *creates *variable A *with value 1,
> > > persists
> > > > it
> > > > > > to
> > > > > > > IGNITE cache and creates *variable B *and sends it to* server2.
> > > *The
> > > > > > > latests receives *variable B*, do some logic with it and stores
> > to
> > > > > > IGNITE.
> > > > > > > All the work made by both servers must be fulfilled in *one*
> > > > > transaction.
> > > > > > > Because we need all information done, or nothing(rollbacked).
> The
> > > > > > scenario
> > > > > > > is managed by orchestrator.
> > > > > > >
> > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin <
> > > > sergi.vlady...@gmail.com
> > > > > >:
> > > > > > >
> > > > > > > > Ok, it is not a business case, it is your wrong solution for
> > it.
> > > > > > > > Lets try again, what is the business case?
> > > > > > > >
> > > > > > > > Sergi
> > > > > > > >
> > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > alkuznetsov...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > The case is the following, One starts transaction in one
> > node,
> > > > and
> > > > > > > commit
> > > > > > > > > this transaction in another jvm node(or rollback it
> > remotely).
> > > > > > > > >
> > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin <
> > > > > > sergi.vlady...@gmail.com
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Because even if you make it work for some simplistic
> > > scenario,
> > > > > get
> > > > > > > > ready
> > > > > > > > > to
> > > > > > > > > > write many fault tolerance tests and make sure that you
> TXs
> > > > work
> > > > > > > > > gracefully
> > > > > > > > > > in all modes in case of crashes. Also make sure that we
> do
> > > not
> > > > > have
> > > > > > > any
> > > > > > > > > > performance drops after all your changes in existing
> > > > benchmarks.
> > > > > > All
> > > > > > > in
> > > > > > > > > all
> > > > > > > > > > I don't believe these conditions will be met and your
> > > > > contribution
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > accepted.
> > > > > > > > > >
> > > > > > > > > > Better solution to what problem? Sending TX to another
> > node?
> > > > The
> > > > > > > > problem
> > > > > > > > > > statement itself is already wrong. What business case you
> > are
> > > > > > trying
> > > > > > > to
> > > > > > > > > > solve? I'm sure everything you need can be done in a much
> > > more
> > > > > > simple
> > > > > > > > and
> > > > > > > > > > efficient way at the application level.
> > > > > > > > > >
> > > > > > > > > > Sergi
> > > > > > > > > >
> > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > > > alkuznetsov...@gmail.com
> > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Why wrong ? You know the better solution?
> > > > > > > > > > >
> > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin <
> > > > > > > > sergi.vlady...@gmail.com
> > > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Just serializing TX object and deserializing it on
> > > another
> > > > > node
> > > > > > > is
> > > > > > > > > > > > meaningless, because other nodes participating in the
> > TX
> > > > have
> > > > > > to
> > > > > > > > know
> > > > > > > > > > > about
> > > > > > > > > > > > the new coordinator. This will require protocol
> > changes,
> > > we
> > > > > > > > > definitely
> > > > > > > > > > > will
> > > > > > > > > > > > have fault tolerance and performance issues. IMO the
> > > whole
> > > > > idea
> > > > > > > is
> > > > > > > > > > wrong
> > > > > > > > > > > > and it makes no sense to waste time on it.
> > > > > > > > > > > >
> > > > > > > > > > > > Sergi
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > IgniteTransactionState implememntation contains
> > > > > > IgniteTxEntry's
> > > > > > > > > which
> > > > > > > > > > > is
> > > > > > > > > > > > > supposed to be transferable
> > > > > > > > > > > > >
> > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan <
> > > > > > > > > > dsetrak...@apache.org
> > > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > It sounds a little scary to me that we are
> passing
> > > > > > > transaction
> > > > > > > > > > > objects
> > > > > > > > > > > > > > around. Such object may contain all sorts of
> Ignite
> > > > > > context.
> > > > > > > If
> > > > > > > > > > some
> > > > > > > > > > > > data
> > > > > > > > > > > > > > needs to be passed across, we should create a
> > special
> > > > > > > transfer
> > > > > > > > > > object
> > > > > > > > > > > > in
> > > > > > > > > > > > > > this case.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > D.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY
> KUZNETSOV
> > <
> > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > well, there a couple of issues preventing
> > > transaction
> > > > > > > > > proceeding.
> > > > > > > > > > > > > > > At first, After transaction serialization and
> > > > > > > deserialization
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > > remote
> > > > > > > > > > > > > > > server, there is no txState. So im going to put
> > it
> > > in
> > > > > > > > > > > > > > > writeExternal()\readExternal()
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The last one is Deserialized transaction lacks
> of
> > > > > shared
> > > > > > > > cache
> > > > > > > > > > > > context
> > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must
> > be
> > > > > > injected
> > > > > > > > by
> > > > > > > > > > > > > > > GridResourceProcessor ?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV
> <
> > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > while starting and continuing transaction in
> > > > > different
> > > > > > > jvms
> > > > > > > > > in
> > > > > > > > > > > run
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > serialization exception in writeExternalMeta
> :
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Override public void
> > writeExternal(ObjectOutput
> > > > out)
> > > > > > > > throws
> > > > > > > > > > > > > > IOException
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > writeExternalMeta(out);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > some meta is cannot be serialized.
> > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey
> Goncharuk <
> > > > > > > > > > > > > > > alexey.goncha...@gmail.com
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Aleksey,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think I am starting to get what you want,
> > but I
> > > > > have
> > > > > > a
> > > > > > > > few
> > > > > > > > > > > > > concerns:
> > > > > > > > > > > > > > > > - What is the API for the proposed change?
> In
> > > your
> > > > > > test,
> > > > > > > > you
> > > > > > > > > > > pass
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > instance of transaction created on ignite(0)
> to
> > > the
> > > > > > > ignite
> > > > > > > > > > > instance
> > > > > > > > > > > > > > > > ignite(1). This is obviously not possible in
> a
> > > > truly
> > > > > > > > > > distributed
> > > > > > > > > > > > > > > > (multi-jvm) environment.
> > > > > > > > > > > > > > > > - How will you synchronize cache update
> actions
> > > and
> > > > > > > > > transaction
> > > > > > > > > > > > > commit?
> > > > > > > > > > > > > > > > Say, you have one node that decided to
> commit,
> > > but
> > > > > > > another
> > > > > > > > > node
> > > > > > > > > > > is
> > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > writing within this transaction. How do you
> > make
> > > > sure
> > > > > > > that
> > > > > > > > > two
> > > > > > > > > > > > nodes
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not call commit() and rollback()
> > simultaneously?
> > > > > > > > > > > > > > > > - How do you make sure that either commit()
> or
> > > > > > > rollback()
> > > > > > > > is
> > > > > > > > > > > > called
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > originator failed?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов <
> > > > > > > > > > somefire...@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial
> > understanding
> > > > was
> > > > > > > that
> > > > > > > > > > > > > transferring
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > tx
> > > > > > > > > > > > > > > > > ownership from one node to another will be
> > > > happened
> > > > > > > > > > > automatically
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > originating node is gone down.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY
> KUZNETSOV
> > <
> > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple
> > > > > threads,
> > > > > > > > nodes,
> > > > > > > > > > > > > > jvms(soon).
> > > > > > > > > > > > > > > > So
> > > > > > > > > > > > > > > > > > every node is able to rollback, or commit
> > > > common
> > > > > > > > > > > transaction.It
> > > > > > > > > > > > > > > turned
> > > > > > > > > > > > > > > > > up i
> > > > > > > > > > > > > > > > > > need to transfer tx between nodes in
> order
> > to
> > > > > > commit
> > > > > > > > > > > > transaction
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > different node(in the same jvm).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey
> > > Goncharuk <
> > > > > > > > > > > > > > > > > alexey.goncha...@gmail.com
> > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Aleksey,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of
> > > > > > transferring
> > > > > > > > of
> > > > > > > > > tx
> > > > > > > > > > > > > > ownership
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > one node to another? My initial
> > > understanding
> > > > > was
> > > > > > > > that
> > > > > > > > > > you
> > > > > > > > > > > > want
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > to update keys in a transaction from
> > > multiple
> > > > > > > threads
> > > > > > > > > in
> > > > > > > > > > > > > > parallel.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > --AG
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY
> > > KUZNETSOV
> > > > <
> > > > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in
> > one
> > > > > node,
> > > > > > > and
> > > > > > > > > > > > continued
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > > > > > one.
> > > > > > > > > > > > > > > > > > > > The following test describes my idea:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0);
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions =
> > > > > > > > > > ignite1.transactions();
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache =
> > > > > > > > > > > > > ignite1.getOrCreateCache("
> > > > > > > > > > > > > > > > > > > > testCache");
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Transaction tx =
> transactions.txStart(
> > > > > > > concurrency,
> > > > > > > > > > > > > isolation);
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > cache.put("key1", 1);
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > cache.put("key2", 2);
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > tx.stop();
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut =
> > > > > > > > > > > > GridTestUtils.runAsync(()
> > > > > > > > > > > > > > ->
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > IgniteTransactions ts =
> > > > > > > > ignite(1).transactions();
> > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx());
> > > > > > > > > > > > > > > > > > > > Assert.assertEquals(
> > > > > > > TransactionState.STOPPED,
> > > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > > > > ts.txStart(tx);
> > > > > > > > > > > > > > > > > > > >
> > > > > > Assert.assertEquals(TransactionState.ACTIVE,
> > > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > > > > cache.put("key3", 3);
> > > > > > > > > > > > > > > > > > > >
> > > > Assert.assertTrue(cache.remove("key2"));
> > > > > > > > > > > > > > > > > > > > tx.commit();
> > > > > > > > > > > > > > > > > > > > return true;
> > > > > > > > > > > > > > > > > > > > });
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > fut.get();
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Assert.assertEquals(
> > > > > TransactionState.COMMITTED,
> > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1,
> > > > > > > > > (long)cache.get("key1"));
> > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3,
> > > > > > > > > (long)cache.get("key3"));
> > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache.
> > > > > containsKey("key2"));
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just
> > > rebind
> > > > > *tx*
> > > > > > > to
> > > > > > > > > > > current
> > > > > > > > > > > > > > > thread:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) {
> > > > > > > > > > > > > > > > > > > > TransactionProxyImpl
> > > transactionProxy =
> > > > > > > > > > > > > > > > (TransactionProxyImpl)tx;
> > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx(
> > > > > transactionProxy.tx());
> > > > > > > > > > > > > > > > > > > > transactionProxy.
> > > bindToCurrentThread();
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter
> > *threadMap*
> > > > so
> > > > > > that
> > > > > > > > it
> > > > > > > > > > > binds
> > > > > > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > > > > > to current thread.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > How do u think about it ?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis
> > Magda <
> > > > > > > > > > > dma...@apache.org
> > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Alexey,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Please share the rational behind
> this
> > > and
> > > > > the
> > > > > > > > > > thoughts,
> > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > > ideas
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > have in mind.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > —
> > > > > > > > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM,
> ALEKSEY
> > > > > > > KUZNETSOV <
> > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed
> > > > > > transaction
> > > > > > > > > which
> > > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > started
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > > node, and continued at other one.
> > Has
> > > > > > anybody
> > > > > > > > > > > thoughts
> > > > > > > > > > > > on
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > ?
> > > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > *Best Regards,*
> > > > > > > > > > >
> > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > *Best Regards,*
> > > > > > > > >
> > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *Best Regards,*
> > > > > > >
> > > > > > > *Kuznetsov Aleksey*
> > > > > > >
> > > > > >
> > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > > >
> > > >
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>
