Hi, Igniters! I 've made implementation of transactions of non-single coordinator. Here you can start transaction in one thread and commit it in another thread. Take a look on it. Give your thoughts on it.
https://github.com/voipp/ignite/pull/10/commits/3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin <sergi.vlady...@gmail.com>: > You know better, go ahead! :) > > Sergi > > 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>: > > > we've discovered several problems regarding your "accumulation" > > approach.These are > > > > 1. perfomance issues when transfering data from temporary cache to > > permanent one. Keep in mind big deal of concurent transactions in > > Service > > commiter > > 2. extreme memory load when keeping temporary cache in memory > > 3. As long as user is not acquainted with ignite, working with cache > > must be transparent for him. Keep this in mind. User's node can > evaluate > > logic with no transaction at all, so we should deal with both types of > > execution flow : transactional and non-transactional.Another one > > problem is > > transaction id support at the user node. We would have handled all > this > > issues and many more. > > 4. we cannot pessimistically lock entity. > > > > As a result, we decided to move on building distributed transaction. We > put > > aside your "accumulation" approach until we realize how to solve > > difficulties above . > > > > чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <sergi.vlady...@gmail.com>: > > > > > The problem "How to run millions of entities, and millions of > operations > > on > > > a single Pentium3" is out of scope here. Do the math, plan capacity > > > reasonably. > > > > > > Sergi > > > > > > 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com > >: > > > > > > > hmm, If we have millions of entities, and millions of operations, > would > > > not > > > > this approache lead to memory overflow and perfomance degradation > > > > > > > > чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < > sergi.vlady...@gmail.com > > >: > > > > > > > > > 1. Actually you have to check versions on all the values you have > > read > > > > > during the tx. > > > > > > > > > > For example if we have [k1 => v1, k2 => v2] and do: > > > > > > > > > > put(k1, get(k2) + 5) > > > > > > > > > > We have to remember the version for k2. This logic can be > relatively > > > > easily > > > > > encapsulated in a framework atop of Ignite. You need to implement > one > > > to > > > > > make all this stuff usable. > > > > > > > > > > 2. I suggest to avoid any locking here, because you easily will end > > up > > > > with > > > > > deadlocks. If you do not have too frequent updates for your keys, > > > > > optimistic approach will work just fine. > > > > > > > > > > Theoretically in the Committer Service you can start a thread for > the > > > > > lifetime of the whole distributed transaction, take a lock on the > key > > > > using > > > > > IgniteCache.lock(K key) before executing any Services, wait for all > > the > > > > > services to complete, execute optimistic commit in the same thread > > > while > > > > > keeping this lock and then release it. Notice that all the Ignite > > > > > transactions inside of all Services must be optimistic here to be > > able > > > to > > > > > read this locked key. > > > > > > > > > > But again I do not recommend you using this approach until you > have a > > > > > reliable deadlock avoidance scheme. > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > > alkuznetsov...@gmail.com > > > >: > > > > > > > > > > > Yeah, now i got it. > > > > > > There are some doubts on this approach > > > > > > 1) During optimistic commit phase, when you assure no one altered > > the > > > > > > original values, you must check versions of other dependent keys. > > How > > > > > could > > > > > > we obtain those keys(in an automative manner, of course) ? > > > > > > 2) How could we lock a key before some Service A introduce > changes? > > > So > > > > no > > > > > > other service is allowed to change this key-value?(sort of > > > pessimistic > > > > > > blocking) > > > > > > May be you know some implementations of such approach ? > > > > > > > > > > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > > > > alkuznetsov...@gmail.com > > > > > >: > > > > > > > > > > > > > Thank you very much for help. I will answer later. > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > > > > sergi.vlady...@gmail.com > > > > > >: > > > > > > > > > > > > > > All the services do not update key in place, but only generate > > new > > > > keys > > > > > > > augmented by otx and store the updated value in the same cache > + > > > > > remember > > > > > > > the keys and versions participating in the transaction in some > > > > separate > > > > > > > atomic cache. > > > > > > > > > > > > > > Follow this sequence of changes applied to cache contents by > each > > > > > > Service: > > > > > > > > > > > > > > Initial cache contents: > > > > > > > [k1 => v1] > > > > > > > [k2 => v2] > > > > > > > [k3 => v3] > > > > > > > > > > > > > > Cache contents after Service A: > > > > > > > [k1 => v1] > > > > > > > [k2 => v2] > > > > > > > [k3 => v3] > > > > > > > [k1x => v1a] > > > > > > > [k2x => v2a] > > > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate > > atomic > > > > > cache > > > > > > > > > > > > > > Cache contents after Service B: > > > > > > > [k1 => v1] > > > > > > > [k2 => v2] > > > > > > > [k3 => v3] > > > > > > > [k1x => v1a] > > > > > > > [k2x => v2ab] > > > > > > > [k3x => v3b] > > > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > > > separate > > > > > > > atomic cache > > > > > > > > > > > > > > Finally the Committer Service takes this map of updated keys > and > > > > their > > > > > > > versions from some separate atomic cache, starts Ignite > > transaction > > > > and > > > > > > > replaces all the values for k* keys to values taken from k*x > > keys. > > > > The > > > > > > > successful result must be the following: > > > > > > > > > > > > > > [k1 => v1a] > > > > > > > [k2 => v2ab] > > > > > > > [k3 => v3b] > > > > > > > [k1x => v1a] > > > > > > > [k2x => v2ab] > > > > > > > [k3x => v3b] > > > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > > > separate > > > > > > > atomic cache > > > > > > > > > > > > > > But Committer Service also has to check that no one updated the > > > > > original > > > > > > > values before us, because otherwise we can not give any > > > > serializability > > > > > > > guarantee for these distributed transactions. Here we may need > to > > > > check > > > > > > not > > > > > > > only versions of the updated keys, but also versions of any > other > > > > keys > > > > > > end > > > > > > > result depends on. > > > > > > > > > > > > > > After that Committer Service has to do a cleanup (may be > outside > > of > > > > the > > > > > > > committing tx) to come to the following final state: > > > > > > > > > > > > > > [k1 => v1a] > > > > > > > [k2 => v2ab] > > > > > > > [k3 => v3b] > > > > > > > > > > > > > > Makes sense? > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > > > > alkuznetsov...@gmail.com > > > > > >: > > > > > > > > > > > > > > > - what do u mean by saying " > > > > > > > > *in a single transaction checks value versions for all the > old > > > > values > > > > > > > > and replaces them with calculated new ones *"? Every time > > you > > > > > > change > > > > > > > > value(in some service), you store it to *some special > atomic > > > > > cache* > > > > > > , > > > > > > > so > > > > > > > > when all services ceased working, Service commiter got a > > > values > > > > > with > > > > > > > the > > > > > > > > last versions. > > > > > > > > - After "*does cleanup of temporary keys and values*" > > Service > > > > > > commiter > > > > > > > > persists them into permanent store, isn't it ? > > > > > > > > - I cant grasp your though, you say "*in case of version > > > > mismatch > > > > > or > > > > > > > TX > > > > > > > > timeout just rollbacks*". But what versions would it > match? > > > > > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > > > > > sergi.vlady...@gmail.com > > > > > > >: > > > > > > > > > > > > > > > > > Ok, here is what you actually need to implement at the > > > > application > > > > > > > level. > > > > > > > > > > > > > > > > > > Lets say we have to call 2 services in the following order: > > > > > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] > to > > > > [k1 > > > > > => > > > > > > > > v1a, > > > > > > > > > k2 => v2a] > > > > > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] > to > > > [k2 > > > > > => > > > > > > > > v2ab, > > > > > > > > > k3 => v3b] > > > > > > > > > > > > > > > > > > The change > > > > > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > > > > > > must happen in a single transaction. > > > > > > > > > > > > > > > > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > > > > > > > > > > > Each cache key must have a field `otx`, which is a unique > > > > > > orchestrator > > > > > > > TX > > > > > > > > > identifier - it must be a parameter passed to all the > > services. > > > > If > > > > > > > `otx` > > > > > > > > is > > > > > > > > > set to some value it means that it is an intermediate key > and > > > is > > > > > > > visible > > > > > > > > > only inside of some transaction, for the finalized key > `otx` > > > must > > > > > be > > > > > > > > null - > > > > > > > > > it means the key is committed and visible for everyone. > > > > > > > > > > > > > > > > > > Each cache value must have a field `ver` which is a version > > of > > > > that > > > > > > > > value. > > > > > > > > > > > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use > > > UUID. > > > > > > > > > > > > > > > > > > Workflow is the following: > > > > > > > > > > > > > > > > > > Orchestrator starts the distributed transaction with `otx` > = > > x > > > > and > > > > > > > passes > > > > > > > > > this parameter to all the services. > > > > > > > > > > > > > > > > > > Service A: > > > > > > > > > - does some computations > > > > > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > > > > > > where > > > > > > > > > Za - left time from max Orchestrator TX duration > > > after > > > > > > > Service > > > > > > > > A > > > > > > > > > end > > > > > > > > > k1x, k2x - new temporary keys with field `otx` = > x > > > > > > > > > v2a has updated version `ver` > > > > > > > > > - returns a set of updated keys and all the old versions > to > > > the > > > > > > > > > orchestrator > > > > > > > > > or just stores it in some special atomic cache like > > > > > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > > > > > > > > > > > Service B: > > > > > > > > > - retrieves the updated value k2x => v2a because it knows > > > `otx` > > > > = > > > > > x > > > > > > > > > - does computations > > > > > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > > > > > > - updates the set of updated keys like [x => (k1 -> ver1, > k2 > > > -> > > > > > > ver2, > > > > > > > k3 > > > > > > > > > -> ver3)] TTL = Zb > > > > > > > > > > > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > > > > > > - takes all the updated keys and versions for `otx` = x > > > > > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > > > > > > - in a single transaction checks value versions for all > the > > > old > > > > > > values > > > > > > > > > and replaces them with calculated new ones > > > > > > > > > - does cleanup of temporary keys and values > > > > > > > > > - in case of version mismatch or TX timeout just rollbacks > > and > > > > > > signals > > > > > > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > > > > > > > > > > > PROFIT!! > > > > > > > > > > > > > > > > > > This approach even allows you to run independent parts of > the > > > > graph > > > > > > in > > > > > > > > > parallel (with TX transfer you will always run only one at > a > > > > time). > > > > > > > Also > > > > > > > > it > > > > > > > > > does not require inventing any special fault tolerance > > technics > > > > > > because > > > > > > > > > Ignite caches are already fault tolerant and all the > > > intermediate > > > > > > > results > > > > > > > > > are virtually invisible and stored with TTL, thus in case > of > > > any > > > > > > crash > > > > > > > > you > > > > > > > > > will not have inconsistent state or garbage. > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > alkuznetsov...@gmail.com > > > > > > > >: > > > > > > > > > > > > > > > > > > > Okay, we are open for proposals on business task. I mean, > > we > > > > can > > > > > > make > > > > > > > > use > > > > > > > > > > of some other thing, not distributed transaction. Not > > > > transaction > > > > > > > yet. > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > > > > > voze...@gridgain.com > > > > > > >: > > > > > > > > > > > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > > > > > > mentioned, > > > > > > > > the > > > > > > > > > > > problem is far more complex, than simply passing TX > state > > > > over > > > > > a > > > > > > > > wire. > > > > > > > > > > Most > > > > > > > > > > > probably a kind of coordinator will be required still > to > > > > manage > > > > > > all > > > > > > > > > kinds > > > > > > > > > > > of failures. This task should be started with clean > > design > > > > > > proposal > > > > > > > > > > > explaining how we handle all these concurrent events. > And > > > > only > > > > > > > then, > > > > > > > > > when > > > > > > > > > > > we understand all implications, we should move to > > > development > > > > > > > stage. > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > > > > > > alkuznetsov...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > Right > > > > > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > > > > > > sergi.vlady...@gmail.com > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some > > > > > predefined > > > > > > > > graph > > > > > > > > > of > > > > > > > > > > > > > distributed services to be invoked, calls them by > > some > > > > kind > > > > > > of > > > > > > > > RPC > > > > > > > > > > and > > > > > > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible > > for > > > > > > > managing > > > > > > > > > > > business > > > > > > > > > > > > > > scenarios flows. Many nodes are involved in > > > scenarios. > > > > > They > > > > > > > > > > exchange > > > > > > > > > > > > data > > > > > > > > > > > > > > and folow one another. If you acquinted with BPMN > > > > > > framework, > > > > > > > so > > > > > > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > > > > > > sergi.vlady...@gmail.com > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing > from > > > > > > Microsoft > > > > > > > or > > > > > > > > > > your > > > > > > > > > > > > > custom > > > > > > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers > > which > > > > > > fulfills > > > > > > > > > > custom > > > > > > > > > > > > > logic. > > > > > > > > > > > > > > > > This servers compound oriented graph (BPMN > > > process) > > > > > > which > > > > > > > > > > > > controlled > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > Orchestrator. > > > > > > > > > > > > > > > > For instance, *server1 *creates *variable A > > > *with > > > > > > value > > > > > > > 1, > > > > > > > > > > > > persists > > > > > > > > > > > > > it > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > IGNITE cache and creates *variable B *and > sends > > > it > > > > > to* > > > > > > > > > server2. > > > > > > > > > > > > *The > > > > > > > > > > > > > > > > latests receives *variable B*, do some logic > > with > > > > it > > > > > > and > > > > > > > > > stores > > > > > > > > > > > to > > > > > > > > > > > > > > > IGNITE. > > > > > > > > > > > > > > > > All the work made by both servers must be > > > fulfilled > > > > > in > > > > > > > > *one* > > > > > > > > > > > > > > transaction. > > > > > > > > > > > > > > > > Because we need all information done, or > > > > > > > > nothing(rollbacked). > > > > > > > > > > The > > > > > > > > > > > > > > > scenario > > > > > > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > > > > > > sergi.vlady...@gmail.com > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your > > wrong > > > > > > > solution > > > > > > > > > for > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts > > > > transaction > > > > > > in > > > > > > > > one > > > > > > > > > > > node, > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > this transaction in another jvm node(or > > > > rollback > > > > > it > > > > > > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi > > Vladykin < > > > > > > > > > > > > > > > sergi.vlady...@gmail.com > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for > some > > > > > > > simplistic > > > > > > > > > > > > scenario, > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > ready > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > write many fault tolerance tests and > make > > > > sure > > > > > > that > > > > > > > > you > > > > > > > > > > TXs > > > > > > > > > > > > > work > > > > > > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > > > > > > in all modes in case of crashes. Also > > make > > > > sure > > > > > > > that > > > > > > > > we > > > > > > > > > > do > > > > > > > > > > > > not > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > performance drops after all your > changes > > in > > > > > > > existing > > > > > > > > > > > > > benchmarks. > > > > > > > > > > > > > > > All > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > I don't believe these conditions will > be > > > met > > > > > and > > > > > > > your > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? > Sending > > TX > > > > to > > > > > > > > another > > > > > > > > > > > node? > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > > > > statement itself is already wrong. What > > > > > business > > > > > > > case > > > > > > > > > you > > > > > > > > > > > are > > > > > > > > > > > > > > > trying > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > solve? I'm sure everything you need can > > be > > > > done > > > > > > in > > > > > > > a > > > > > > > > > much > > > > > > > > > > > > more > > > > > > > > > > > > > > > simple > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better > > solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi > > > > Vladykin < > > > > > > > > > > > > > > > > > sergi.vlady...@gmail.com > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and > > > > > deserializing > > > > > > it > > > > > > > > on > > > > > > > > > > > > another > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > meaningless, because other nodes > > > > > > participating > > > > > > > in > > > > > > > > > the > > > > > > > > > > > TX > > > > > > > > > > > > > have > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > the new coordinator. This will > > require > > > > > > protocol > > > > > > > > > > > changes, > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > have fault tolerance and > performance > > > > > issues. > > > > > > > IMO > > > > > > > > > the > > > > > > > > > > > > whole > > > > > > > > > > > > > > idea > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > > > > > > and it makes no sense to waste time > > on > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY > > > > > KUZNETSOV > > > > > > < > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState > > > implememntation > > > > > > > contains > > > > > > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, > > Dmitriy > > > > > > > Setrakyan > > > > > > > > < > > > > > > > > > > > > > > > > > > > dsetrak...@apache.org > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me > > that > > > > we > > > > > > are > > > > > > > > > > passing > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > > > > > around. Such object may contain > > all > > > > > sorts > > > > > > > of > > > > > > > > > > Ignite > > > > > > > > > > > > > > > context. > > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > > > needs to be passed across, we > > > should > > > > > > > create a > > > > > > > > > > > special > > > > > > > > > > > > > > > > transfer > > > > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 > AM, > > > > > ALEKSEY > > > > > > > > > > KUZNETSOV > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of > issues > > > > > > preventing > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > > > > > > At first, After transaction > > > > > > serialization > > > > > > > > and > > > > > > > > > > > > > > > > deserialization > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > > > > > > server, there is no txState. > So > > > im > > > > > > going > > > > > > > to > > > > > > > > > put > > > > > > > > > > > it > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > > > > > > transaction > > > > > > > > > lacks > > > > > > > > > > of > > > > > > > > > > > > > > shared > > > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > > > > > > field at > TransactionProxyImpl. > > > > > Perhaps, > > > > > > > it > > > > > > > > > must > > > > > > > > > > > be > > > > > > > > > > > > > > > injected > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, > > > > ALEKSEY > > > > > > > > > KUZNETSOV > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and > continuing > > > > > > > transaction > > > > > > > > > in > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > jvms > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > > > > > > writeExternalMeta > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > > > > > > writeExternal(ObjectOutput > > > > > > > > > > > > > out) > > > > > > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be > > > > serialized. > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в > 17:25, > > > > Alexey > > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > > alexey.goncha...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to > get > > > what > > > > > you > > > > > > > > want, > > > > > > > > > > > but I > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > > > > > > - What is the API for the > > > > proposed > > > > > > > > change? > > > > > > > > > > In > > > > > > > > > > > > your > > > > > > > > > > > > > > > test, > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > > > instance of transaction > > created > > > > on > > > > > > > > > ignite(0) > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > > > ignite > > > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > > > > ignite(1). This is > obviously > > > not > > > > > > > possible > > > > > > > > > in > > > > > > > > > > a > > > > > > > > > > > > > truly > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > > > > > > - How will you synchronize > > > cache > > > > > > update > > > > > > > > > > actions > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > > > > > > Say, you have one node that > > > > decided > > > > > > to > > > > > > > > > > commit, > > > > > > > > > > > > but > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > > > writing within this > > > transaction. > > > > > How > > > > > > do > > > > > > > > you > > > > > > > > > > > make > > > > > > > > > > > > > sure > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > > not call commit() and > > > rollback() > > > > > > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > > > > > > - How do you make sure > that > > > > either > > > > > > > > > commit() > > > > > > > > > > or > > > > > > > > > > > > > > > > rollback() > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 > > > > Дмитрий > > > > > > > Рябов > > > > > > > > < > > > > > > > > > > > > > > > > > > > somefire...@gmail.com > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my > > > > initial > > > > > > > > > > > understanding > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > > > > ownership from one node > to > > > > > another > > > > > > > will > > > > > > > > > be > > > > > > > > > > > > > happened > > > > > > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > > > > > originating node is gone > > > down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 > GMT+03:00 > > > > > ALEKSEY > > > > > > > > > > KUZNETSOV > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span > > > transaction > > > > > on > > > > > > > > > multiple > > > > > > > > > > > > > > threads, > > > > > > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > > > > every node is able to > > > > rollback, > > > > > > or > > > > > > > > > commit > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx > > between > > > > > nodes > > > > > > > in > > > > > > > > > > order > > > > > > > > > > > to > > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > different node(in the > > same > > > > > jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в > > > 15:20, > > > > > > Alexey > > > > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > > > > > alexey.goncha...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you > > > want a > > > > > > > concept > > > > > > > > > of > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > > > one node to another? > My > > > > > initial > > > > > > > > > > > > understanding > > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a > > > > > transaction > > > > > > > > from > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > threads > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 > > > GMT+03:00 > > > > > > > ALEKSEY > > > > > > > > > > > > KUZNETSOV > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider > > > > transaction > > > > > > > > started > > > > > > > > > in > > > > > > > > > > > one > > > > > > > > > > > > > > node, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following test > > > > > describes > > > > > > my > > > > > > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = > > > > ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > > > > transactions = > > > > > > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, > > > > > Integer> > > > > > > > > cache > > > > > > > > > = > > > > > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > > > > > > transactions.txStart( > > > > > > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", > 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", > 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > > > > > > fut = > > > > > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > ts = > > > > > > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", > > > 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > > > > *ts.txStart(...)* > > > > > > we > > > > > > > > just > > > > > > > > > > > > rebind > > > > > > > > > > > > > > *tx* > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > > > > > > txStart(Transaction > > > > > > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm > > > ().reopenTx( > > > > > > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > *reopenTx* > > we > > > > > alter > > > > > > > > > > > *threadMap* > > > > > > > > > > > > > so > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think > about > > > it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. > в > > > > 22:38, > > > > > > > Denis > > > > > > > > > > > Magda < > > > > > > > > > > > > > > > > > > > > dma...@apache.org > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the > > > > rational > > > > > > > > behind > > > > > > > > > > this > > > > > > > > > > > > and > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, > > at > > > > 3:19 > > > > > > AM, > > > > > > > > > > ALEKSEY > > > > > > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > alkuznetsov...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im > > > designing > > > > > > > > > distributed > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and > > continued > > > > at > > > > > > > other > > > > > > > > > one. > > > > > > > > > > > Has > > > > > > > > > > > > > > > anybody > > > > > > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov > > Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > -- *Best Regards,* *Kuznetsov Aleksey*
