I'll keep the common pool then, thank you.
Public pool is the "work-horse of the Compute Grid" [1], so it does not
seem to fit here at all.

[1]
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning#public-pool

On Thu, Apr 15, 2021 at 2:18 PM Stanislav Lukyanov <stanlukya...@gmail.com>
wrote:

> Giving this another thought - I'm kinda torn on this myself now, as I like
> you argument that a chain of multiple (GG and not GG) continuations should
> execute in the same pool.
> This would probably be easier to understand for a beginning or
> intermediate Ignite user, which is the majority.
> Anyway, I'll leave to you to decide.
>
> > On 15 Apr 2021, at 11:02, Stanislav Lukyanov <stanlukya...@gmail.com>
> wrote:
> >
> > Hi Pavel,
> >
> > I'd prefer public pool.
> >
> > Thanks,
> > Stan
> >
> >> On 12 Apr 2021, at 20:17, Pavel Tupitsyn <ptupit...@apache.org> wrote:
> >>
> >> Stan,
> >>
> >> Sorry for the late reply (had a vacation).
> >>
> >>> In my ideal world, the users don't configure thread pools, they just
> have
> >> safe default behavior (async execution)
> >>> and a way to make it fast for one particular function (with annotation
> or
> >> anything else).
> >>
> >> I've
> >> added
> testRemoteOperationListenerExecutesOnStripedPoolWhenCustomExecutorIsProvided
> >> to demonstrate this use case.
> >>
> >>
> >>> I'm OK to proceed with the approach you're suggesting if I haven't
> >> convinced you by now
> >>
> >> Are you OK to merge the changes as is (ForkJoinPool#commonPool as the
> >> default executor),
> >> or do we change it to Ignite public pool?
> >>
> >> On Mon, Mar 29, 2021 at 11:09 PM Stanislav Lukyanov <
> stanlukya...@gmail.com>
> >> wrote:
> >>
> >>> But what if I need to have exactly one callback synchronous, and all
> other
> >>> can be asynchronous?
> >>>
> >>> I would separate two cases: an existing user who wants their old
> behavior
> >>> back, and a new user that wants to fine tune their app.
> >>> The existing user needs a global "make it all synchronous" switch.
> >>> The new user should only enable the fast-but-dangerous behavior
> locally,
> >>> exactly where they need it.
> >>>
> >>> In my ideal world, the users don't configure thread pools, they just
> have
> >>> safe default behavior (async execution)
> >>> and a way to make it fast for one particular function (with annotation
> or
> >>> anything else).
> >>> Also, this should work in a similar way for different APIs - so I'm
> trying
> >>> to lay some basis to rework all of these continuous queries and event
> >>> listeners,
> >>> even though they're explicitly mentioned as out of scope for IEP-70.
> >>>
> >>> At the same time, I understand that any change we make now will have
> pros
> >>> and cons, and we can't make it perfect because of compatibility
> reasons.
> >>> I'm OK to proceed with the approach you're suggesting if I haven't
> >>> convinced you by now :)
> >>>
> >>> Thanks,
> >>> Stan
> >>>
> >>>> On 29 Mar 2021, at 22:47, Pavel Tupitsyn <ptupit...@apache.org>
> wrote:
> >>>>
> >>>> Stan,
> >>>>
> >>>> Unfortunately, annotations have a few drawbacks:
> >>>> * Can't configure it globally ("I already use sync callbacks, give me
> >>> back
> >>>> the old behavior in one line")
> >>>> * Can't configure in Spring
> >>>> * Useless in C++ & .NET
> >>>> * You can already specify executor in IgniteFuture#listenAsync, so
> there
> >>>> seems to be no added value
> >>>>
> >>>>> the only value we really expect the user to set in that property is
> >>>> Runnable::run
> >>>> Not really - there are lots of available options [1].
> >>>> Some apps may already have one or more thread pools that can be used
> for
> >>>> continuations.
> >>>>
> >>>>> you can't specify Runnable::run in a Spring XML
> >>>> Good point, but creating a class for that is trivial.
> >>>> We can ship a ready-made class and mention it in the docs for
> simplicity.
> >>>>
> >>>>
> >>>> Globally configurable Executor fits nicely with
> >>>> existing IgniteFuture#listenAsync,
> >>>> not sure why you dislike it.
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>
> https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html
> >>>>
> >>>> On Mon, Mar 29, 2021 at 10:23 PM Stanislav Lukyanov <
> >>> stanlukya...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Thought about this some more.
> >>>>>
> >>>>> I agree that we need to be able to switch to synchronous listeners
> when
> >>>>> it's critical for performance.
> >>>>> However, I don't like to introduce an Executor property for that. In
> >>> fact,
> >>>>> the only value
> >>>>> we really expect the user to set in that property is Runnable::run -
> >>> seems
> >>>>> to be an overkill to have accept an Executor for that.
> >>>>> Furthermore, you can't specify Runnable::run in a Spring XML, can
> you?
> >>>>>
> >>>>> I'm thinking that maybe we should go the annotation route here.
> >>>>> Let's introduce an annotation @IgniteSyncCallback. It's the same as
> >>>>> @IgniteAsyncCallback but reverse :)
> >>>>> If a listener is annotated like that then we execute it in the same
> >>>>> thread; by default, we execute in the public pool.
> >>>>> We can also reuse the same annotation for all other callbacks we
> have in
> >>>>> the system - right now, the callbacks are a mix of sync and async
> >>> behavior,
> >>>>> and we could transition all APIs to use async by default and enforce
> >>> sync
> >>>>> callbacks when the annotation is used.
> >>>>> @IgniteAsyncCallback should eventually be deprecated.
> >>>>>
> >>>>> WDYT?
> >>>>>
> >>>>> Thanks,
> >>>>> Stan
> >>>>>
> >>>>>> On 29 Mar 2021, at 14:09, Pavel Tupitsyn <ptupit...@apache.org>
> wrote:
> >>>>>>
> >>>>>> Stan,
> >>>>>>
> >>>>>> I'm ok with using public pool by default, but we need a way to
> restore
> >>>>> the
> >>>>>> old behavior, do you agree?
> >>>>>> I think we should keep the new IgniteConfiguration property.
> >>>>>>
> >>>>>> On Fri, Mar 26, 2021 at 2:12 PM Alexei Scherbakov <
> >>>>>> alexey.scherbak...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Pavel,
> >>>>>>>
> >>>>>>> Dedicated pool looks safer and more manageable to me. Make sure the
> >>>>> threads
> >>>>>>> in the pool are lazily started and stopped if not used for some
> time.
> >>>>>>>
> >>>>>>> Because I have no more real arguments against the change, I
> suggest to
> >>>>>>> proceed with this approach.
> >>>>>>>
> >>>>>>> чт, 25 мар. 2021 г. в 22:16, Pavel Tupitsyn <ptupit...@apache.org
> >:
> >>>>>>>
> >>>>>>>> Alexei,
> >>>>>>>>
> >>>>>>>>> we already have ways to control a listener's behavior
> >>>>>>>> No, we don't have a way to fix current broken and dangerous
> behavior
> >>>>>>>> globally.
> >>>>>>>> You should not expect the user to fix every async call manually.
> >>>>>>>>
> >>>>>>>>> commonPool can alter existing deployments in unpredictable ways,
> >>>>>>>>> if commonPool is heavily used for other purposes
> >>>>>>>> Common pool resizes dynamically to accommodate the load [1]
> >>>>>>>> What do you think about Stan's suggestion to use our public pool
> >>>>> instead?
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html
> >>>>>>>>
> >>>>>>>> On Thu, Mar 25, 2021 at 10:10 PM Pavel Tupitsyn <
> >>> ptupit...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>>> I don't agree that the code isn't related to Ignite - it is
> >>> something
> >>>>>>>>> that the user does via Ignite API
> >>>>>>>>>
> >>>>>>>>> This is a misconception. When you write general-purpose async
> code,
> >>> it
> >>>>>>>>> looks like this:
> >>>>>>>>>
> >>>>>>>>> myClass.fooAsync()
> >>>>>>>>> .chain(igniteCache.putAsync)
> >>>>>>>>> .chain(myClass.barAsync)
> >>>>>>>>> .chain(...)
> >>>>>>>>>
> >>>>>>>>> And so on, you jump from one continuation to another.
> >>>>>>>>> You don't think about this as "I use Ignite API to run my
> >>>>>>> continuation",
> >>>>>>>>> this is just another async call among hundreds of others.
> >>>>>>>>>
> >>>>>>>>> And you don't want 1 of 20 libraries that you use to have
> "special
> >>>>>>> needs"
> >>>>>>>>> like Ignite does right now.
> >>>>>>>>>
> >>>>>>>>> I know Java is late to the async party and not everyone is used
> to
> >>>>> this
> >>>>>>>>> mindset,
> >>>>>>>>> but the situation changes, more and more code bases go async all
> the
> >>>>>>> way,
> >>>>>>>>> use CompletionStage everywhere, etc.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> If we go with the public pool - no additional options needed.
> >>>>>>>>>
> >>>>>>>>> I guess public pool should work.
> >>>>>>>>> However, I would prefer to keep using commonPool, which is
> >>> recommended
> >>>>>>>> for
> >>>>>>>>> a general purpose like this.
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 25, 2021 at 3:56 PM Alexei Scherbakov <
> >>>>>>>>> alexey.scherbak...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Pavel,
> >>>>>>>>>>
> >>>>>>>>>> The change still looks a bit risky to me, because the default
> >>>>> executor
> >>>>>>>> is
> >>>>>>>>>> set to commonPool and can alter existing deployments in
> >>> unpredictable
> >>>>>>>>>> ways,
> >>>>>>>>>> if commonPool is heavily used for other purposes.
> >>>>>>>>>>
> >>>>>>>>>> Runnable::run usage is not obvious as well and should be
> properly
> >>>>>>>>>> documented as a way to return to old behavior.
> >>>>>>>>>>
> >>>>>>>>>> I'm not sure we need it in 2.X for the reasons above - we
> already
> >>>>> have
> >>>>>>>>>> ways
> >>>>>>>>>> to control a listener's behavior - it's a matter of good
> >>>>> documentation
> >>>>>>>> to
> >>>>>>>>>> me.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> чт, 25 мар. 2021 г. в 15:33, Pavel Tupitsyn <
> ptupit...@apache.org
> >>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Alexei,
> >>>>>>>>>>>
> >>>>>>>>>>>> Sometimes it's more desirable to execute the listener in the
> same
> >>>>>>>>>> thread
> >>>>>>>>>>>> It's up to the user to decide.
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, we give users a choice to configure the executor as
> >>>>>>> Runnable::run
> >>>>>>>>>> and
> >>>>>>>>>>> use the same thread if needed.
> >>>>>>>>>>> However, it should not be the default behavior as explained
> above
> >>>>>>> (bad
> >>>>>>>>>>> usability, unexpected major issues).
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Mar 25, 2021 at 3:06 PM Alexei Scherbakov <
> >>>>>>>>>>> alexey.scherbak...@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Pavel,
> >>>>>>>>>>>>
> >>>>>>>>>>>> While I understand the issue and overall agree with you, I'm
> >>>>>>> against
> >>>>>>>>>> the
> >>>>>>>>>>>> execution of listeners in separate thread pool by default.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sometimes it's more desirable to execute the listener in the
> same
> >>>>>>>>>> thread,
> >>>>>>>>>>>> for example if it's some lightweight closure.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It's up to the user to decide.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think the IgniteFuture.listen method should be properly
> >>>>>>> documented
> >>>>>>>>>> to
> >>>>>>>>>>>> avoid execution of cluster operations or any other potentially
> >>>>>>>>>> blocking
> >>>>>>>>>>>> operations inside the listener.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Otherwise listenAsync should be used.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> чт, 25 мар. 2021 г. в 14:04, Pavel Tupitsyn <
> >>> ptupit...@apache.org
> >>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Stan,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We have thread pools dedicated for specific purposes, like
> cache
> >>>>>>>>>>>> (striped),
> >>>>>>>>>>>>> compute (pub), query, etc
> >>>>>>>>>>>>> As I understand it, the reason here is to limit the number of
> >>>>>>>>>> threads
> >>>>>>>>>>>>> dedicated to a given subsystem.
> >>>>>>>>>>>>> For example, Compute may be overloaded with work, but Cache
> and
> >>>>>>>>>>> Discovery
> >>>>>>>>>>>>> will keep going.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This is different from async continuations, which are
> arbitrary
> >>>>>>>> user
> >>>>>>>>>>>> code.
> >>>>>>>>>>>>> So what is the benefit of having a new user pool for
> arbitrary
> >>>>>>>> code
> >>>>>>>>>>> that
> >>>>>>>>>>>> is
> >>>>>>>>>>>>> probably not related to Ignite at all?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Mar 25, 2021 at 1:31 PM <stanlukya...@gmail.com>
> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Pavel,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This is a great work, fully agree with the overall idea and
> >>>>>>>>>> approach.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> However, I have some reservations about the API. We sure do
> >>>>>>>> have a
> >>>>>>>>>>> lot
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>> async stuff in the system, and I would suggest to stick to
> the
> >>>>>>>>>> usual
> >>>>>>>>>>>>> design
> >>>>>>>>>>>>>> - create a separate thread pool, add a single property to
> >>>>>>>> control
> >>>>>>>>>> the
> >>>>>>>>>>>>> size
> >>>>>>>>>>>>>> of the pool.
> >>>>>>>>>>>>>> Alternatively, we may consider using public pool for that.
> >>>>>>> May I
> >>>>>>>>>> ask
> >>>>>>>>>>> if
> >>>>>>>>>>>>>> there is an example use case which doesn’t work with public
> >>>>>>>> pool?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> For .NET, agree that we should follow the rules and APIs of
> >>>>>>> the
> >>>>>>>>>>>> platform,
> >>>>>>>>>>>>>> so the behavior might slightly differ.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Stan
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 24 Mar 2021, at 09:52, Pavel Tupitsyn <
> >>>>>>>> ptupit...@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Igniters, since there are no more comments and/or review
> >>>>>>>>>> feedback,
> >>>>>>>>>>>>>>> I'm going to merge the changes by the end of the week.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Mar 22, 2021 at 10:37 PM Pavel Tupitsyn <
> >>>>>>>>>>>> ptupit...@apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ready for review:
> >>>>>>>>>>>>>>>> https://github.com/apache/ignite/pull/8870
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sun, Mar 21, 2021 at 8:09 PM Pavel Tupitsyn <
> >>>>>>>>>>>> ptupit...@apache.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Simple benchmark added - see JmhCacheAsyncListenBenchmark
> >>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>> PR.
> >>>>>>>>>>>>>>>>> There is a 6-8% drop (1 client, 2 servers, 1 machine, int
> >>>>>>>>>>> key/val).
> >>>>>>>>>>>>>>>>> I expect this difference to become barely observable on
> >>>>>>>>>>> real-world
> >>>>>>>>>>>>>>>>> workloads.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Mar 18, 2021 at 12:35 PM Pavel Tupitsyn <
> >>>>>>>>>>>>> ptupit...@apache.org>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Denis,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For a reproducer, please see
> >>>>>>>>>>>> CacheAsyncContinuationExecutorTest.java
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the linked PoC [1]
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://github.com/apache/ignite/pull/8870/files#diff-c788c12013622093df07d8f628a6e8c5fb0c15007f0787f2d459dbb3e377fc5aR54
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Mar 18, 2021 at 1:58 AM Raymond Wilson <
> >>>>>>>>>>>>>>>>>> raymond_wil...@trimble.com> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We implemented the ContinueWith() suggestion from
> Pavel,
> >>>>>>>>>> and it
> >>>>>>>>>>>>> works
> >>>>>>>>>>>>>>>>>>> well
> >>>>>>>>>>>>>>>>>>> so far in testing, though we do not have data to
> support
> >>>>>>>> if
> >>>>>>>>>>> there
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> not a performance penalty in our use case..
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> To lend another vote to the 'Don't do continuations on
> >>>>>>> the
> >>>>>>>>>>>> striped
> >>>>>>>>>>>>>>>>>>> thread
> >>>>>>>>>>>>>>>>>>> pool' line of thinking: Deadlocking is an issue as is
> >>>>>>>>>>> starvation.
> >>>>>>>>>>>>> In
> >>>>>>>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>>>>> ways starvation is more insidious because by the time
> >>>>>>>> things
> >>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>> working
> >>>>>>>>>>>>>>>>>>> the cause and effect distance may be large.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I appreciate the documentation does make statements
> >>>>>>> about
> >>>>>>>>>> not
> >>>>>>>>>>>>>> performing
> >>>>>>>>>>>>>>>>>>> cache operations in a continuation due to deadlock
> >>>>>>>>>>> possibilities,
> >>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> statement does not reveal why this is an issue. It is
> >>>>>>>> less a
> >>>>>>>>>>> case
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> async cache operation being followed by some other
> cache
> >>>>>>>>>>>> operation
> >>>>>>>>>>>>>> (an
> >>>>>>>>>>>>>>>>>>> immediate issue), and more a general case of the
> >>>>>>>>>> continuation
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> application logic using a striped pool thread in a way
> >>>>>>>> that
> >>>>>>>>>>> might
> >>>>>>>>>>>>>> mean
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> thread is never given back - it's now just a piece of
> >>>>>>> the
> >>>>>>>>>>>>> application
> >>>>>>>>>>>>>>>>>>> infrastructure until some other application activity
> >>>>>>>>>> schedules
> >>>>>>>>>>>> away
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>> that thread (eg: by ContinueWith or some other async
> >>>>>>>>>> operation
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> application code that releases the thread). To be fair,
> >>>>>>>>>> beyond
> >>>>>>>>>>>>>>>>>>> structures
> >>>>>>>>>>>>>>>>>>> like ContinueWith(), it is not obvious how that
> >>>>>>>> continuation
> >>>>>>>>>>>> thread
> >>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>> be handed back. This will be the same behaviour for
> >>>>>>>>>> dedicated
> >>>>>>>>>>>>>>>>>>> continuation
> >>>>>>>>>>>>>>>>>>> pools, but with far less risk in the absence of
> >>>>>>>>>> ContinueWith()
> >>>>>>>>>>>>>>>>>>> constructs.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> In the .Net world this is becoming more of an issue as
> >>>>>>>> fewer
> >>>>>>>>>>> .Net
> >>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>> cases
> >>>>>>>>>>>>>>>>>>> outside of UI bother with synchronization contexts by
> >>>>>>>>>> default.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Raymond.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Mar 18, 2021 at 9:56 AM Valentin Kulichenko <
> >>>>>>>>>>>>>>>>>>> valentin.kuliche...@gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Denis,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I think Pavel's main point is that behavior is
> >>>>>>>>>> unpredictable.
> >>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>>>>> AFAIK, putAsync can be executed in the main thread
> >>>>>>>> instead
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> striped
> >>>>>>>>>>>>>>>>>>>> pool thread if the operation is local. The listener
> can
> >>>>>>>>>> also
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> executed in
> >>>>>>>>>>>>>>>>>>>> the main thread - this happens if the future is
> >>>>>>> completed
> >>>>>>>>>>> prior
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> listener
> >>>>>>>>>>>>>>>>>>>> invocation (this is actually quite possible in the
> unit
> >>>>>>>>>> test
> >>>>>>>>>>>>>>>>>>> environment
> >>>>>>>>>>>>>>>>>>>> causing the test to pass). Finally, I'm pretty sure
> >>>>>>> there
> >>>>>>>>>> are
> >>>>>>>>>>>> many
> >>>>>>>>>>>>>>>>>>> cases
> >>>>>>>>>>>>>>>>>>>> when a deadlock does not occur right away, but instead
> >>>>>>> it
> >>>>>>>>>> will
> >>>>>>>>>>>>>> reveal
> >>>>>>>>>>>>>>>>>>>> itself under high load due to thread starvation. The
> >>>>>>>> latter
> >>>>>>>>>>> type
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>>>> is the most dangerous because they are often
> reproduced
> >>>>>>>>>> only
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> production.
> >>>>>>>>>>>>>>>>>>>> Finally, there are performance considerations as well
> -
> >>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>> operations
> >>>>>>>>>>>>>>>>>>>> and listeners share the same fixed-sized pool which
> can
> >>>>>>>>>> have
> >>>>>>>>>>>>>> negative
> >>>>>>>>>>>>>>>>>>>> effects.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I'm OK with the change. Although, it might be better
> to
> >>>>>>>>>>>> introduce
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>> fixed-sized pool instead of ForkJoinPool for
> listeners,
> >>>>>>>>>> simply
> >>>>>>>>>>>>>>>>>>> because this
> >>>>>>>>>>>>>>>>>>>> is the approach taken throughout the project. But this
> >>>>>>> is
> >>>>>>>>>> up
> >>>>>>>>>>> to
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> debate.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -Val
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Mar 17, 2021 at 11:31 AM Denis Garus <
> >>>>>>>>>>>> garus....@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Pavel,
> >>>>>>>>>>>>>>>>>>>>> I tried this:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> @Test
> >>>>>>>>>>>>>>>>>>>>> public void test() throws Exception {
> >>>>>>>>>>>>>>>>>>>>> IgniteCache<Integer, String> cache =
> >>>>>>>>>>>>>>>>>>>>> startGrid().getOrCreateCache("test_cache");
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> cache.putAsync(1, "one").listen(f ->
> >>>>>>> cache.replace(1,
> >>>>>>>>>>>> "two"));
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> assertEquals("two", cache.get(1));
> >>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> and this test is green.
> >>>>>>>>>>>>>>>>>>>>> I believe that an user can make listener that leads
> to
> >>>>>>>>>>>> deadlock,
> >>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>> the example in the IEP does not reflect this.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ср, 17 мар. 2021 г. в 17:36, Вячеслав Коптилин <
> >>>>>>>>>>>>>>>>>>> slava.kopti...@gmail.com
> >>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Pavel,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Not a good excuse really. We have a usability
> >>>>>>> problem,
> >>>>>>>>>> you
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> admit
> >>>>>>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>>>> Fair enough. I agree that this is a usability issue,
> >>>>>>>> but
> >>>>>>>>>> I
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> doubts
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> the proposed approach to overcome it is the best
> one.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Documentation won't help - no one is going to read
> >>>>>>> the
> >>>>>>>>>>>> Javadoc
> >>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>> trivial method like putAsync
> >>>>>>>>>>>>>>>>>>>>>> That is sad... However, I don't think that this is a
> >>>>>>>>>> strong
> >>>>>>>>>>>>>>>>>>> argument
> >>>>>>>>>>>>>>>>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> This is just my opinion. Let's see what other
> >>>>>>> community
> >>>>>>>>>>>> members
> >>>>>>>>>>>>>>>>>>> have to
> >>>>>>>>>>>>>>>>>>>>>> say.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>> S.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ср, 17 мар. 2021 г. в 17:01, Pavel Tupitsyn <
> >>>>>>>>>>>>> ptupit...@apache.org
> >>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> the user should use the right API
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Not a good excuse really. We have a usability
> >>>>>>> problem,
> >>>>>>>>>> you
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> admit
> >>>>>>>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>>>>> "The brakes did not work on your car - too bad, you
> >>>>>>>>>> should
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> known
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>> on Sundays only your left foot is allowed on the
> >>>>>>>> pedal"
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> This particular use case is too intricate.
> >>>>>>>>>>>>>>>>>>>>>>> Even when you know about that, it is difficult to
> >>>>>>>> decide
> >>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>> can run
> >>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>> the striped pool,
> >>>>>>>>>>>>>>>>>>>>>>> and what can't. It is too easy to forget.
> >>>>>>>>>>>>>>>>>>>>>>> And most people don't know, even among Ignite
> >>>>>>>>>> developers.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Documentation won't help - no one is going to read
> >>>>>>> the
> >>>>>>>>>>>> Javadoc
> >>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>> trivial method like putAsync.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> So I propose to have a safe default.
> >>>>>>>>>>>>>>>>>>>>>>> Then document the performance tuning opportunity on
> >>>>>>>> [1].
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Think about how many users abandon a product
> because
> >>>>>>>> it
> >>>>>>>>>>>>>>>>>>> mysteriously
> >>>>>>>>>>>>>>>>>>>>>>> crashes and hangs.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 17, 2021 at 4:21 PM Вячеслав Коптилин <
> >>>>>>>>>>>>>>>>>>>>>>> slava.kopti...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Pavel,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Well, I think that the user should use the right
> >>>>>>> API
> >>>>>>>>>>> instead
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>> introducing
> >>>>>>>>>>>>>>>>>>>>>>>> uncontested overhead for everyone.
> >>>>>>>>>>>>>>>>>>>>>>>> For instance, the code that is provided by IEP can
> >>>>>>>>>> changed
> >>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>> follows:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> IgniteFuture fut = cache.putAsync(1, 1);
> >>>>>>>>>>>>>>>>>>>>>>>> fut.listenAync(f -> {
> >>>>>>>>>>>>>>>>>>>>>>>> // Executes on Striped pool and deadlocks.
> >>>>>>>>>>>>>>>>>>>>>>>> cache.replace(1, 2);
> >>>>>>>>>>>>>>>>>>>>>>>> }, ForkJoinPool.commonPool());
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Of course, it does not mean that this fact should
> >>>>>>> not
> >>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> properly
> >>>>>>>>>>>>>>>>>>>>>>>> documented.
> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps, I am missing something.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> S.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> ср, 17 мар. 2021 г. в 16:01, Pavel Tupitsyn <
> >>>>>>>>>>>>>>>>>>> ptupit...@apache.org
> >>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Slava,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Your suggestion is to keep things as is and
> >>>>>>> discard
> >>>>>>>>>> the
> >>>>>>>>>>>> IEP,
> >>>>>>>>>>>>>>>>>>>> right?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> this can lead to significant overhead
> >>>>>>>>>>>>>>>>>>>>>>>>> Yes, there is some overhead, but the cost of
> >>>>>>>>>> accidentally
> >>>>>>>>>>>>>>>>>>>> starving
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> striped pool is worse,
> >>>>>>>>>>>>>>>>>>>>>>>>> not to mention the deadlocks.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I believe that we should favor correctness over
> >>>>>>>>>>> performance
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>> case.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 17, 2021 at 3:34 PM Вячеслав Коптилин
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> slava.kopti...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Well, the specified method already exists :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> /**
> >>>>>>>>>>>>>>>>>>>>>>>>>> * Registers listener closure to be
> >>>>>>>> asynchronously
> >>>>>>>>>>>>>>>>>>> notified
> >>>>>>>>>>>>>>>>>>>>>>>> whenever
> >>>>>>>>>>>>>>>>>>>>>>>>>> future completes.
> >>>>>>>>>>>>>>>>>>>>>>>>>> * Closure will be processed in specified
> >>>>>>>>>> executor.
> >>>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>>>>>>>>> * @param lsnr Listener closure to register.
> >>>>>>>>>> Cannot
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> {@code
> >>>>>>>>>>>>>>>>>>>>>>>> null}.
> >>>>>>>>>>>>>>>>>>>>>>>>>> * @param exec Executor to run listener.
> >>>>>>> Cannot
> >>>>>>>> be
> >>>>>>>>>>>>>>>>>>> {@code
> >>>>>>>>>>>>>>>>>>>>>> null}.
> >>>>>>>>>>>>>>>>>>>>>>>>>> */
> >>>>>>>>>>>>>>>>>>>>>>>>>> public void listenAsync(IgniteInClosure<?
> >>>>>>> super
> >>>>>>>>>>>>>>>>>>>>>> IgniteFuture<V>>
> >>>>>>>>>>>>>>>>>>>>>>>>> lsnr,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Executor exec);
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> S.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ср, 17 мар. 2021 г. в 15:25, Вячеслав Коптилин <
> >>>>>>>>>>>>>>>>>>>>>>>> slava.kopti...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Pavel,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I took a look at your IEP and pool request. I
> >>>>>>> have
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> following
> >>>>>>>>>>>>>>>>>>>>>>>>>> concerns.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> First of all, this change breaks the contract
> of
> >>>>>>>>>>>>>>>>>>>>>>>>>> IgniteFuture#listen(lsnr)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> /**
> >>>>>>>>>>>>>>>>>>>>>>>>>>> * Registers listener closure to be
> >>>>>>>>>> asynchronously
> >>>>>>>>>>>>>>>>>>>> notified
> >>>>>>>>>>>>>>>>>>>>>>>>> whenever
> >>>>>>>>>>>>>>>>>>>>>>>>>>> future completes.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> * Closure will be processed in thread that
> >>>>>>>>>>>>>>>>>>> completes
> >>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>> future
> >>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>> (if future already
> >>>>>>>>>>>>>>>>>>>>>>>>>>> * completed) immediately in current thread.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>>>>>>>>>> * @param lsnr Listener closure to register.
> >>>>>>>>>> Cannot
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> {@code
> >>>>>>>>>>>>>>>>>>>>>>>>> null}.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> */
> >>>>>>>>>>>>>>>>>>>>>>>>>>> public void listen(IgniteInClosure<? super
> >>>>>>>>>>>>>>>>>>>> IgniteFuture<V>>
> >>>>>>>>>>>>>>>>>>>>>>>> lsnr);
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> In your pull request, the listener is always
> >>>>>>>>>> called
> >>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> specified
> >>>>>>>>>>>>>>>>>>>>>>>>>>> thread pool (which is fork-join by default)
> >>>>>>>>>>>>>>>>>>>>>>>>>>> even though the future is already completed
> >>>>>>> at
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> moment
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> listen
> >>>>>>>>>>>>>>>>>>>>>>>>>>> method is called.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> In my opinion, this can lead to significant
> >>>>>>>>>>>>>>>>>>> overhead -
> >>>>>>>>>>>>>>>>>>>>>>> submission
> >>>>>>>>>>>>>>>>>>>>>>>>>>> requires acquiring a lock and notifying a pool
> >>>>>>>>>> thread.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> It seems to me, that we should not change the
> >>>>>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>> behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> However, thread pool executor can be added as
> an
> >>>>>>>>>>>>>>>>>>> optional
> >>>>>>>>>>>>>>>>>>>>>> parameter
> >>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> listen() method as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>    public void listen(IgniteInClosure<?
> >>>>>>> super
> >>>>>>>>>>>>>>>>>>>>>> IgniteFuture<V>>
> >>>>>>>>>>>>>>>>>>>>>>>>> lsnr,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Executor exec);
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> S.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> пн, 15 мар. 2021 г. в 19:24, Pavel Tupitsyn <
> >>>>>>>>>>>>>>>>>>>>>> ptupit...@apache.org
> >>>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Please review the IEP [1] and let me know your
> >>>>>>>>>>>>>>>>>>> thoughts.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-70%3A+Async+Continuation+Executor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> <http://www.trimble.com/>
> >>>>>>>>>>>>>>>>>>> Raymond Wilson
> >>>>>>>>>>>>>>>>>>> Solution Architect, Civil Construction Software Systems
> >>>>>>>>>> (CCSS)
> >>>>>>>>>>>>>>>>>>> 11 Birmingham Drive | Christchurch, New Zealand
> >>>>>>>>>>>>>>>>>>> raymond_wil...@trimble.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best regards,
> >>>>>>>>>>>> Alexei Scherbakov
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>> Best regards,
> >>>>>>>>>> Alexei Scherbakov
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Alexei Scherbakov
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
>
>

Reply via email to