we've discovered several problems regarding your "accumulation"
approach.These are
1. perfomance issues when transfering data from temporary cache to
permanent one. Keep in mind big deal of concurent transactions in Service
commiter
2. extreme memory load when keeping temporary cache in memory
3. As long as user is not acquainted with ignite, working with cache
must be transparent for him. Keep this in mind. User's node can evaluate
logic with no transaction at all, so we should deal with both types of
execution flow : transactional and non-transactional.Another one problem is
transaction id support at the user node. We would have handled all this
issues and many more.
4. we cannot pessimistically lock entity.
As a result, we decided to move on building distributed transaction. We put
aside your "accumulation" approach until we realize how to solve
difficulties above .
чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <sergi.vlady...@gmail.com>:
> The problem "How to run millions of entities, and millions of operations on
> a single Pentium3" is out of scope here. Do the math, plan capacity
> reasonably.
>
> Sergi
>
> 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>:
>
> > hmm, If we have millions of entities, and millions of operations, would
> not
> > this approache lead to memory overflow and perfomance degradation
> >
> > чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin <sergi.vlady...@gmail.com>:
> >
> > > 1. Actually you have to check versions on all the values you have read
> > > during the tx.
> > >
> > > For example if we have [k1 => v1, k2 => v2] and do:
> > >
> > > put(k1, get(k2) + 5)
> > >
> > > We have to remember the version for k2. This logic can be relatively
> > easily
> > > encapsulated in a framework atop of Ignite. You need to implement one
> to
> > > make all this stuff usable.
> > >
> > > 2. I suggest to avoid any locking here, because you easily will end up
> > with
> > > deadlocks. If you do not have too frequent updates for your keys,
> > > optimistic approach will work just fine.
> > >
> > > Theoretically in the Committer Service you can start a thread for the
> > > lifetime of the whole distributed transaction, take a lock on the key
> > using
> > > IgniteCache.lock(K key) before executing any Services, wait for all the
> > > services to complete, execute optimistic commit in the same thread
> while
> > > keeping this lock and then release it. Notice that all the Ignite
> > > transactions inside of all Services must be optimistic here to be able
> to
> > > read this locked key.
> > >
> > > But again I do not recommend you using this approach until you have a
> > > reliable deadlock avoidance scheme.
> > >
> > > Sergi
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com
> >:
> > >
> > > > Yeah, now i got it.
> > > > There are some doubts on this approach
> > > > 1) During optimistic commit phase, when you assure no one altered the
> > > > original values, you must check versions of other dependent keys. How
> > > could
> > > > we obtain those keys(in an automative manner, of course) ?
> > > > 2) How could we lock a key before some Service A introduce changes?
> So
> > no
> > > > other service is allowed to change this key-value?(sort of
> pessimistic
> > > > blocking)
> > > > May be you know some implementations of such approach ?
> > > >
> > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV <
> > alkuznetsov...@gmail.com
> > > >:
> > > >
> > > > > Thank you very much for help. I will answer later.
> > > > >
> > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin <
> > sergi.vlady...@gmail.com
> > > >:
> > > > >
> > > > > All the services do not update key in place, but only generate new
> > keys
> > > > > augmented by otx and store the updated value in the same cache +
> > > remember
> > > > > the keys and versions participating in the transaction in some
> > separate
> > > > > atomic cache.
> > > > >
> > > > > Follow this sequence of changes applied to cache contents by each
> > > > Service:
> > > > >
> > > > > Initial cache contents:
> > > > > [k1 => v1]
> > > > > [k2 => v2]
> > > > > [k3 => v3]
> > > > >
> > > > > Cache contents after Service A:
> > > > > [k1 => v1]
> > > > > [k2 => v2]
> > > > > [k3 => v3]
> > > > > [k1x => v1a]
> > > > > [k2x => v2a]
> > > > >
> > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic
> > > cache
> > > > >
> > > > > Cache contents after Service B:
> > > > > [k1 => v1]
> > > > > [k2 => v2]
> > > > > [k3 => v3]
> > > > > [k1x => v1a]
> > > > > [k2x => v2ab]
> > > > > [k3x => v3b]
> > > > >
> > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some
> > separate
> > > > > atomic cache
> > > > >
> > > > > Finally the Committer Service takes this map of updated keys and
> > their
> > > > > versions from some separate atomic cache, starts Ignite transaction
> > and
> > > > > replaces all the values for k* keys to values taken from k*x keys.
> > The
> > > > > successful result must be the following:
> > > > >
> > > > > [k1 => v1a]
> > > > > [k2 => v2ab]
> > > > > [k3 => v3b]
> > > > > [k1x => v1a]
> > > > > [k2x => v2ab]
> > > > > [k3x => v3b]
> > > > >
> > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some
> > separate
> > > > > atomic cache
> > > > >
> > > > > But Committer Service also has to check that no one updated the
> > > original
> > > > > values before us, because otherwise we can not give any
> > serializability
> > > > > guarantee for these distributed transactions. Here we may need to
> > check
> > > > not
> > > > > only versions of the updated keys, but also versions of any other
> > keys
> > > > end
> > > > > result depends on.
> > > > >
> > > > > After that Committer Service has to do a cleanup (may be outside of
> > the
> > > > > committing tx) to come to the following final state:
> > > > >
> > > > > [k1 => v1a]
> > > > > [k2 => v2ab]
> > > > > [k3 => v3b]
> > > > >
> > > > > Makes sense?
> > > > >
> > > > > Sergi
> > > > >
> > > > >
> > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV <
> > alkuznetsov...@gmail.com
> > > >:
> > > > >
> > > > > > - what do u mean by saying "
> > > > > > *in a single transaction checks value versions for all the old
> > values
> > > > > > and replaces them with calculated new ones *"? Every time you
> > > > change
> > > > > > value(in some service), you store it to *some special atomic
> > > cache*
> > > > ,
> > > > > so
> > > > > > when all services ceased working, Service commiter got a
> values
> > > with
> > > > > the
> > > > > > last versions.
> > > > > > - After "*does cleanup of temporary keys and values*" Service
> > > > commiter
> > > > > > persists them into permanent store, isn't it ?
> > > > > > - I cant grasp your though, you say "*in case of version
> > mismatch
> > > or
> > > > > TX
> > > > > > timeout just rollbacks*". But what versions would it match?
> > > > > >
> > > > > >
> > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin <
> > > sergi.vlady...@gmail.com
> > > > >:
> > > > > >
> > > > > > > Ok, here is what you actually need to implement at the
> > application
> > > > > level.
> > > > > > >
> > > > > > > Lets say we have to call 2 services in the following order:
> > > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to
> > [k1
> > > =>
> > > > > > v1a,
> > > > > > > k2 => v2a]
> > > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to
> [k2
> > > =>
> > > > > > v2ab,
> > > > > > > k3 => v3b]
> > > > > > >
> > > > > > > The change
> > > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ]
> > > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ]
> > > > > > > must happen in a single transaction.
> > > > > > >
> > > > > > >
> > > > > > > Optimistic protocol to solve this:
> > > > > > >
> > > > > > > Each cache key must have a field `otx`, which is a unique
> > > > orchestrator
> > > > > TX
> > > > > > > identifier - it must be a parameter passed to all the services.
> > If
> > > > > `otx`
> > > > > > is
> > > > > > > set to some value it means that it is an intermediate key and
> is
> > > > > visible
> > > > > > > only inside of some transaction, for the finalized key `otx`
> must
> > > be
> > > > > > null -
> > > > > > > it means the key is committed and visible for everyone.
> > > > > > >
> > > > > > > Each cache value must have a field `ver` which is a version of
> > that
> > > > > > value.
> > > > > > >
> > > > > > > For both fields (`otx` and `ver`) the safest way is to use
> UUID.
> > > > > > >
> > > > > > > Workflow is the following:
> > > > > > >
> > > > > > > Orchestrator starts the distributed transaction with `otx` = x
> > and
> > > > > passes
> > > > > > > this parameter to all the services.
> > > > > > >
> > > > > > > Service A:
> > > > > > > - does some computations
> > > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za
> > > > > > > where
> > > > > > > Za - left time from max Orchestrator TX duration
> after
> > > > > Service
> > > > > > A
> > > > > > > end
> > > > > > > k1x, k2x - new temporary keys with field `otx` = x
> > > > > > > v2a has updated version `ver`
> > > > > > > - returns a set of updated keys and all the old versions to
> the
> > > > > > > orchestrator
> > > > > > > or just stores it in some special atomic cache like
> > > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za
> > > > > > >
> > > > > > > Service B:
> > > > > > > - retrieves the updated value k2x => v2a because it knows
> `otx`
> > =
> > > x
> > > > > > > - does computations
> > > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb
> > > > > > > - updates the set of updated keys like [x => (k1 -> ver1, k2
> ->
> > > > ver2,
> > > > > k3
> > > > > > > -> ver3)] TTL = Zb
> > > > > > >
> > > > > > > Service Committer (may be embedded into Orchestrator):
> > > > > > > - takes all the updated keys and versions for `otx` = x
> > > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)]
> > > > > > > - in a single transaction checks value versions for all the
> old
> > > > values
> > > > > > > and replaces them with calculated new ones
> > > > > > > - does cleanup of temporary keys and values
> > > > > > > - in case of version mismatch or TX timeout just rollbacks and
> > > > signals
> > > > > > > to Orchestrator to restart the job with new `otx`
> > > > > > >
> > > > > > > PROFIT!!
> > > > > > >
> > > > > > > This approach even allows you to run independent parts of the
> > graph
> > > > in
> > > > > > > parallel (with TX transfer you will always run only one at a
> > time).
> > > > > Also
> > > > > > it
> > > > > > > does not require inventing any special fault tolerance technics
> > > > because
> > > > > > > Ignite caches are already fault tolerant and all the
> intermediate
> > > > > results
> > > > > > > are virtually invisible and stored with TTL, thus in case of
> any
> > > > crash
> > > > > > you
> > > > > > > will not have inconsistent state or garbage.
> > > > > > >
> > > > > > > Sergi
> > > > > > >
> > > > > > >
> > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > alkuznetsov...@gmail.com
> > > > > >:
> > > > > > >
> > > > > > > > Okay, we are open for proposals on business task. I mean, we
> > can
> > > > make
> > > > > > use
> > > > > > > > of some other thing, not distributed transaction. Not
> > transaction
> > > > > yet.
> > > > > > > >
> > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >:
> > > > > > > >
> > > > > > > > > IMO the use case makes sense. However, as Sergi already
> > > > mentioned,
> > > > > > the
> > > > > > > > > problem is far more complex, than simply passing TX state
> > over
> > > a
> > > > > > wire.
> > > > > > > > Most
> > > > > > > > > probably a kind of coordinator will be required still to
> > manage
> > > > all
> > > > > > > kinds
> > > > > > > > > of failures. This task should be started with clean design
> > > > proposal
> > > > > > > > > explaining how we handle all these concurrent events. And
> > only
> > > > > then,
> > > > > > > when
> > > > > > > > > we understand all implications, we should move to
> development
> > > > > stage.
> > > > > > > > >
> > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV <
> > > > > > > > > alkuznetsov...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Right
> > > > > > > > > >
> > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <
> > > > > > > sergi.vlady...@gmail.com
> > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Good! Basically your orchestrator just takes some
> > > predefined
> > > > > > graph
> > > > > > > of
> > > > > > > > > > > distributed services to be invoked, calls them by some
> > kind
> > > > of
> > > > > > RPC
> > > > > > > > and
> > > > > > > > > > > passes the needed parameters between them, right?
> > > > > > > > > > >
> > > > > > > > > > > Sergi
> > > > > > > > > > >
> > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > orchestrator is a custom thing. He is responsible for
> > > > > managing
> > > > > > > > > business
> > > > > > > > > > > > scenarios flows. Many nodes are involved in
> scenarios.
> > > They
> > > > > > > > exchange
> > > > > > > > > > data
> > > > > > > > > > > > and folow one another. If you acquinted with BPMN
> > > > framework,
> > > > > so
> > > > > > > > > > > > orchestrator is like bpmn engine.
> > > > > > > > > > > >
> > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <
> > > > > > > sergi.vlady...@gmail.com
> > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from
> > > > Microsoft
> > > > > or
> > > > > > > > your
> > > > > > > > > > > custom
> > > > > > > > > > > > > in-house software?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sergi
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which
> > > > fulfills
> > > > > > > > custom
> > > > > > > > > > > logic.
> > > > > > > > > > > > > > This servers compound oriented graph (BPMN
> process)
> > > > which
> > > > > > > > > > controlled
> > > > > > > > > > > by
> > > > > > > > > > > > > > Orchestrator.
> > > > > > > > > > > > > > For instance, *server1 *creates *variable A
> *with
> > > > value
> > > > > 1,
> > > > > > > > > > persists
> > > > > > > > > > > it
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > IGNITE cache and creates *variable B *and sends
> it
> > > to*
> > > > > > > server2.
> > > > > > > > > > *The
> > > > > > > > > > > > > > latests receives *variable B*, do some logic with
> > it
> > > > and
> > > > > > > stores
> > > > > > > > > to
> > > > > > > > > > > > > IGNITE.
> > > > > > > > > > > > > > All the work made by both servers must be
> fulfilled
> > > in
> > > > > > *one*
> > > > > > > > > > > > transaction.
> > > > > > > > > > > > > > Because we need all information done, or
> > > > > > nothing(rollbacked).
> > > > > > > > The
> > > > > > > > > > > > > scenario
> > > > > > > > > > > > > > is managed by orchestrator.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin <
> > > > > > > > > > > sergi.vlady...@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong
> > > > > solution
> > > > > > > for
> > > > > > > > > it.
> > > > > > > > > > > > > > > Lets try again, what is the business case?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sergi
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The case is the following, One starts
> > transaction
> > > > in
> > > > > > one
> > > > > > > > > node,
> > > > > > > > > > > and
> > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > this transaction in another jvm node(or
> > rollback
> > > it
> > > > > > > > > remotely).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin <
> > > > > > > > > > > > > sergi.vlady...@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Because even if you make it work for some
> > > > > simplistic
> > > > > > > > > > scenario,
> > > > > > > > > > > > get
> > > > > > > > > > > > > > > ready
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > write many fault tolerance tests and make
> > sure
> > > > that
> > > > > > you
> > > > > > > > TXs
> > > > > > > > > > > work
> > > > > > > > > > > > > > > > gracefully
> > > > > > > > > > > > > > > > > in all modes in case of crashes. Also make
> > sure
> > > > > that
> > > > > > we
> > > > > > > > do
> > > > > > > > > > not
> > > > > > > > > > > > have
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > performance drops after all your changes in
> > > > > existing
> > > > > > > > > > > benchmarks.
> > > > > > > > > > > > > All
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > I don't believe these conditions will be
> met
> > > and
> > > > > your
> > > > > > > > > > > > contribution
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > accepted.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX
> > to
> > > > > > another
> > > > > > > > > node?
> > > > > > > > > > > The
> > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > statement itself is already wrong. What
> > > business
> > > > > case
> > > > > > > you
> > > > > > > > > are
> > > > > > > > > > > > > trying
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > solve? I'm sure everything you need can be
> > done
> > > > in
> > > > > a
> > > > > > > much
> > > > > > > > > > more
> > > > > > > > > > > > > simple
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > efficient way at the application level.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Sergi
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY
> KUZNETSOV
> > <
> > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi
> > Vladykin <
> > > > > > > > > > > > > > > sergi.vlady...@gmail.com
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Just serializing TX object and
> > > deserializing
> > > > it
> > > > > > on
> > > > > > > > > > another
> > > > > > > > > > > > node
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > meaningless, because other nodes
> > > > participating
> > > > > in
> > > > > > > the
> > > > > > > > > TX
> > > > > > > > > > > have
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > the new coordinator. This will require
> > > > protocol
> > > > > > > > > changes,
> > > > > > > > > > we
> > > > > > > > > > > > > > > > definitely
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > have fault tolerance and performance
> > > issues.
> > > > > IMO
> > > > > > > the
> > > > > > > > > > whole
> > > > > > > > > > > > idea
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > and it makes no sense to waste time on
> > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Sergi
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY
> > > KUZNETSOV
> > > > <
> > > > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IgniteTransactionState
> implememntation
> > > > > contains
> > > > > > > > > > > > > IgniteTxEntry's
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > supposed to be transferable
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy
> > > > > Setrakyan
> > > > > > <
> > > > > > > > > > > > > > > > > dsetrak...@apache.org
> > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that
> > we
> > > > are
> > > > > > > > passing
> > > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > > > objects
> > > > > > > > > > > > > > > > > > > > > around. Such object may contain all
> > > sorts
> > > > > of
> > > > > > > > Ignite
> > > > > > > > > > > > > context.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > needs to be passed across, we
> should
> > > > > create a
> > > > > > > > > special
> > > > > > > > > > > > > > transfer
> > > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > this case.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM,
> > > ALEKSEY
> > > > > > > > KUZNETSOV
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues
> > > > preventing
> > > > > > > > > > transaction
> > > > > > > > > > > > > > > > proceeding.
> > > > > > > > > > > > > > > > > > > > > > At first, After transaction
> > > > serialization
> > > > > > and
> > > > > > > > > > > > > > deserialization
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > remote
> > > > > > > > > > > > > > > > > > > > > > server, there is no txState. So
> im
> > > > going
> > > > > to
> > > > > > > put
> > > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal()
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized
> > > > transaction
> > > > > > > lacks
> > > > > > > > of
> > > > > > > > > > > > shared
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > context
> > > > > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl.
> > > Perhaps,
> > > > > it
> > > > > > > must
> > > > > > > > > be
> > > > > > > > > > > > > injected
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27,
> > ALEKSEY
> > > > > > > KUZNETSOV
> > > > > > > > <
> > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > while starting and continuing
> > > > > transaction
> > > > > > > in
> > > > > > > > > > > > different
> > > > > > > > > > > > > > jvms
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > > > serialization exception in
> > > > > > > writeExternalMeta
> > > > > > > > :
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Override public void
> > > > > > > > > writeExternal(ObjectOutput
> > > > > > > > > > > out)
> > > > > > > > > > > > > > > throws
> > > > > > > > > > > > > > > > > > > > > IOException
> > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out);
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be
> > serialized.
> > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25,
> > Alexey
> > > > > > > > Goncharuk <
> > > > > > > > > > > > > > > > > > > > > > alexey.goncha...@gmail.com
> > > > > > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Aleksey,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get
> what
> > > you
> > > > > > want,
> > > > > > > > > but I
> > > > > > > > > > > > have
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > concerns:
> > > > > > > > > > > > > > > > > > > > > > > - What is the API for the
> > proposed
> > > > > > change?
> > > > > > > > In
> > > > > > > > > > your
> > > > > > > > > > > > > test,
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > pass
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > instance of transaction created
> > on
> > > > > > > ignite(0)
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > ignite
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously
> not
> > > > > possible
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > > truly
> > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment.
> > > > > > > > > > > > > > > > > > > > > > > - How will you synchronize
> cache
> > > > update
> > > > > > > > actions
> > > > > > > > > > and
> > > > > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > > > > > commit?
> > > > > > > > > > > > > > > > > > > > > > > Say, you have one node that
> > decided
> > > > to
> > > > > > > > commit,
> > > > > > > > > > but
> > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > node
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > > writing within this
> transaction.
> > > How
> > > > do
> > > > > > you
> > > > > > > > > make
> > > > > > > > > > > sure
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > not call commit() and
> rollback()
> > > > > > > > > simultaneously?
> > > > > > > > > > > > > > > > > > > > > > > - How do you make sure that
> > either
> > > > > > > commit()
> > > > > > > > or
> > > > > > > > > > > > > > rollback()
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > called
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > originator failed?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00
> > Дмитрий
> > > > > Рябов
> > > > > > <
> > > > > > > > > > > > > > > > > somefire...@gmail.com
> > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my
> > initial
> > > > > > > > > understanding
> > > > > > > > > > > was
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > transferring
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > tx
> > > > > > > > > > > > > > > > > > > > > > > > ownership from one node to
> > > another
> > > > > will
> > > > > > > be
> > > > > > > > > > > happened
> > > > > > > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > originating node is gone
> down.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00
> > > ALEKSEY
> > > > > > > > KUZNETSOV
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span
> transaction
> > > on
> > > > > > > multiple
> > > > > > > > > > > > threads,
> > > > > > > > > > > > > > > nodes,
> > > > > > > > > > > > > > > > > > > > > jvms(soon).
> > > > > > > > > > > > > > > > > > > > > > > So
> > > > > > > > > > > > > > > > > > > > > > > > > every node is able to
> > rollback,
> > > > or
> > > > > > > commit
> > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > transaction.It
> > > > > > > > > > > > > > > > > > > > > > turned
> > > > > > > > > > > > > > > > > > > > > > > > up i
> > > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx between
> > > nodes
> > > > > in
> > > > > > > > order
> > > > > > > > > to
> > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > different node(in the same
> > > jvm).
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в
> 15:20,
> > > > Alexey
> > > > > > > > > > Goncharuk <
> > > > > > > > > > > > > > > > > > > > > > > > alexey.goncha...@gmail.com
> > > > > > > > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you
> want a
> > > > > concept
> > > > > > > of
> > > > > > > > > > > > > transferring
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > tx
> > > > > > > > > > > > > > > > > > > > > ownership
> > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > one node to another? My
> > > initial
> > > > > > > > > > understanding
> > > > > > > > > > > > was
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a
> > > transaction
> > > > > > from
> > > > > > > > > > multiple
> > > > > > > > > > > > > > threads
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > parallel.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > --AG
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01
> GMT+03:00
> > > > > ALEKSEY
> > > > > > > > > > KUZNETSOV
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com
> > > > > > > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider
> > transaction
> > > > > > started
> > > > > > > in
> > > > > > > > > one
> > > > > > > > > > > > node,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > continued
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > > > > > > > > > > > > one.
> > > > > > > > > > > > > > > > > > > > > > > > > > > The following test
> > > describes
> > > > my
> > > > > > > idea:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 =
> > ignite(0);
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions
> > > > > transactions =
> > > > > > > > > > > > > > > > > ignite1.transactions();
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String,
> > > Integer>
> > > > > > cache
> > > > > > > =
> > > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache("
> > > > > > > > > > > > > > > > > > > > > > > > > > > testCache");
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx =
> > > > > > > > transactions.txStart(
> > > > > > > > > > > > > > concurrency,
> > > > > > > > > > > > > > > > > > > > isolation);
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1);
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2);
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop();
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > IgniteInternalFuture<Boolean>
> > > > > > fut =
> > > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(()
> > > > > > > > > > > > > > > > > > > > > ->
> > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions
> > ts =
> > > > > > > > > > > > > > > ignite(1).transactions();
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > Assert.assertNull(ts.tx());
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> Assert.assertEquals(
> > > > > > > > > > > > > > TransactionState.STOPPED,
> > > > > > > > > > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx);
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE,
> > > > > > > > > > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3",
> 3);
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > Assert.assertTrue(cache.remove("key2"));
> > > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit();
> > > > > > > > > > > > > > > > > > > > > > > > > > > return true;
> > > > > > > > > > > > > > > > > > > > > > > > > > > });
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get();
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(
> > > > > > > > > > > > TransactionState.COMMITTED,
> > > > > > > > > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Assert.assertEquals((long)1,
> > > > > > > > > > > > > > > > (long)cache.get("key1"));
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Assert.assertEquals((long)3,
> > > > > > > > > > > > > > > > (long)cache.get("key3"));
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> Assert.assertFalse(cache.
> > > > > > > > > > > > containsKey("key2"));
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > In method
> > *ts.txStart(...)*
> > > > we
> > > > > > just
> > > > > > > > > > rebind
> > > > > > > > > > > > *tx*
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > > > thread:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > public void
> > > > txStart(Transaction
> > > > > > > tx) {
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> TransactionProxyImpl
> > > > > > > > > > transactionProxy =
> > > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx;
> > > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm
> ().reopenTx(
> > > > > > > > > > > > transactionProxy.tx());
> > > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy.
> > > > > > > > > > bindToCurrentThread();
> > > > > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we
> > > alter
> > > > > > > > > *threadMap*
> > > > > > > > > > > so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > binds
> > > > > > > > > > > > > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > > > > > > > > > > > > to current thread.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about
> it ?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в
> > 22:38,
> > > > > Denis
> > > > > > > > > Magda <
> > > > > > > > > > > > > > > > > > dma...@apache.org
> > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the
> > rational
> > > > > > behind
> > > > > > > > this
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > thoughts,
> > > > > > > > > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > > > > > > > > > ideas
> > > > > > > > > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > —
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at
> > 3:19
> > > > AM,
> > > > > > > > ALEKSEY
> > > > > > > > > > > > > > KUZNETSOV <
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> alkuznetsov...@gmail.com
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im
> designing
> > > > > > > distributed
> > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > started
> > > > > > > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued
> > at
> > > > > other
> > > > > > > one.
> > > > > > > > > Has
> > > > > > > > > > > > > anybody
> > > > > > > > > > > > > > > > > > thoughts
> > > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > ?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > >
> > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > *Best Regards,*
> > > > > > > > > >
> > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > *Best Regards,*
> > > > > > > >
> > > > > > > > *Kuznetsov Aleksey*
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > >
> > > > > > *Best Regards,*
> > > > > >
> > > > > > *Kuznetsov Aleksey*
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > > >
> > > > --
> > > >
> > > > *Best Regards,*
> > > >
> > > > *Kuznetsov Aleksey*
> > > >
> > >
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>
--
*Best Regards,*
*Kuznetsov Aleksey*
