Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Igor Sapego 
Ivan,

We are not going to adopt C++20 in the near future, as new standards
are not adopted as fast in C++ world and if we'd do that most probably
almost none of our users could use our client.

But yeah, if we'd use coroutines, I'd probably suggest that we only
provide the async API.

Best Regards,
Igor



On Thu, Sep 9, 2021 at 2:29 PM Ivan Daschinsky  wrote:

> Igor, and what about C++20 and coroutines [1]
>
> [1] -- https://en.cppreference.com/w/cpp/language/coroutines
>
> чт, 9 сент. 2021 г. в 14:12, Igor Sapego :
>
> > Well, fortunately we do not provide a C client if you don't consider ODBC
> > as one so we should not think about it. For C++ I believe we should use
> > standard std::future+std::promise for async, but still can provide sync
> > methods,
> > built on top of async methods. There is no continuation problem in C++
> API
> > as
> > std::future do not provide continuation methods :)
> >
> > Now, for the sync/async problem you've mentioned. Can not we solve it by
> > using different thread pools for completion and continuation of futures
> or
> > does
> > CompletableFuture interface not allowing for that?
> >
> > Best Regards,
> > Igor
> >
> >
> > On Thu, Sep 9, 2021 at 1:15 PM Ivan Daschinsky 
> > wrote:
> >
> > > And what about C/C++?  There are so many options for them...
> > >
> > > чт, 9 сент. 2021 г. в 13:00, Pavel Tupitsyn :
> > >
> > > > Talked to Ivan in private, and came up with the following
> per-language
> > > > summary:
> > > >
> > > > * Java: sync and async
> > > > *. NET: only async
> > > > * Python: sync and async
> > > > * JavaScript: only async (sync is not possible)
> > > >
> > > > Thin clients in other languages are to be discussed.
> > > >
> > > > On Thu, Sep 9, 2021 at 11:49 AM Ivan Daschinsky  >
> > > > wrote:
> > > >
> > > > > >> You can mix them easily.
> > > > > This is far from easily (you have already mentioned continuation
> > > > problem),
> > > > > but for i.e. in python it is absolutely not.
> > > > > For kotlin it is a little bit easier, but also not fluent and a
> > little
> > > > bit
> > > > > ugly.
> > > > >
> > > > > чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn  >:
> > > > >
> > > > > > Ivan,
> > > > > >
> > > > > > > Pavel, is it really true, that in .NET sync versions of
> libraries
> > > and
> > > > > > tools
> > > > > > > are completely eliminated?
> > > > > >
> > > > > > Far from being eliminated, but there is a movement in this
> > direction.
> > > > > > For example, HttpClient [1] only provides async variants for most
> > of
> > > > the
> > > > > > methods.
> > > > > >
> > > > > > Exposing sync wrappers for async methods is not recommended [2].
> > > > > > There is a good explanation of potential threading issues there,
> > and
> > > it
> > > > > can
> > > > > > be applied to Java too.
> > > > > >
> > > > > >
> > > > > > > you cannot mix both functions easily
> > > > > >
> > > > > > You can mix them easily.
> > > > > > C#: table.GetAsync(k).Result or
> > > > > table.GetAsync(k).GetAwaiter().GetResult()
> > > > > > (different exception handling)
> > > > > > Java: table.getAsync(k).get()
> > > > > >
> > > > > > The same thing we do in sync wrapper for async method is now in
> the
> > > > user
> > > > > > code.
> > > > > > Which is better for two reasons:
> > > > > > - users won't accidentally use sync API when async API should be
> > used
> > > > > > - when they really have to use sync API, potentially dangerous
> > > behavior
> > > > > is
> > > > > > not hidden
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
> > > > > >
> > > > > > On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > >> 2. In languages with proper async support (async-await,
> etc.),
> > > we
> > > > > can
> > > > > > > skip sync API altogether.
> > > > > > > It sounds pretty strange, as for me. Usually you cannot mix
> both
> > > > > > functions
> > > > > > > easily, it is called blue-red functions problem [1]
> > > > > > > In python you definitely cannot do sync over async, for example
> > > > > > > (principally can, but nobody do that because of mediocre
> > > performance
> > > > of
> > > > > > > that solution). And many developers
> > > > > > > still prefer writing code withou asyncio at all.
> > > > > > >
> > > > > > > Pavel, is it really true, that in .NET sync versions of
> libraries
> > > and
> > > > > > tools
> > > > > > > are completely eliminated?
> > > > > > >
> > > > > > > [1] --
> > > > > > >
> > > > >
> > >
> https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> > > > > > >
> > > > > > > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >:
>

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Ivan Daschinsky 
Igor, and what about C++20 and coroutines [1]

[1] -- https://en.cppreference.com/w/cpp/language/coroutines

чт, 9 сент. 2021 г. в 14:12, Igor Sapego :

> Well, fortunately we do not provide a C client if you don't consider ODBC
> as one so we should not think about it. For C++ I believe we should use
> standard std::future+std::promise for async, but still can provide sync
> methods,
> built on top of async methods. There is no continuation problem in C++ API
> as
> std::future do not provide continuation methods :)
>
> Now, for the sync/async problem you've mentioned. Can not we solve it by
> using different thread pools for completion and continuation of futures or
> does
> CompletableFuture interface not allowing for that?
>
> Best Regards,
> Igor
>
>
> On Thu, Sep 9, 2021 at 1:15 PM Ivan Daschinsky 
> wrote:
>
> > And what about C/C++?  There are so many options for them...
> >
> > чт, 9 сент. 2021 г. в 13:00, Pavel Tupitsyn :
> >
> > > Talked to Ivan in private, and came up with the following per-language
> > > summary:
> > >
> > > * Java: sync and async
> > > *. NET: only async
> > > * Python: sync and async
> > > * JavaScript: only async (sync is not possible)
> > >
> > > Thin clients in other languages are to be discussed.
> > >
> > > On Thu, Sep 9, 2021 at 11:49 AM Ivan Daschinsky 
> > > wrote:
> > >
> > > > >> You can mix them easily.
> > > > This is far from easily (you have already mentioned continuation
> > > problem),
> > > > but for i.e. in python it is absolutely not.
> > > > For kotlin it is a little bit easier, but also not fluent and a
> little
> > > bit
> > > > ugly.
> > > >
> > > > чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn :
> > > >
> > > > > Ivan,
> > > > >
> > > > > > Pavel, is it really true, that in .NET sync versions of libraries
> > and
> > > > > tools
> > > > > > are completely eliminated?
> > > > >
> > > > > Far from being eliminated, but there is a movement in this
> direction.
> > > > > For example, HttpClient [1] only provides async variants for most
> of
> > > the
> > > > > methods.
> > > > >
> > > > > Exposing sync wrappers for async methods is not recommended [2].
> > > > > There is a good explanation of potential threading issues there,
> and
> > it
> > > > can
> > > > > be applied to Java too.
> > > > >
> > > > >
> > > > > > you cannot mix both functions easily
> > > > >
> > > > > You can mix them easily.
> > > > > C#: table.GetAsync(k).Result or
> > > > table.GetAsync(k).GetAwaiter().GetResult()
> > > > > (different exception handling)
> > > > > Java: table.getAsync(k).get()
> > > > >
> > > > > The same thing we do in sync wrapper for async method is now in the
> > > user
> > > > > code.
> > > > > Which is better for two reasons:
> > > > > - users won't accidentally use sync API when async API should be
> used
> > > > > - when they really have to use sync API, potentially dangerous
> > behavior
> > > > is
> > > > > not hidden
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
> > > > >
> > > > > On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > >> 2. In languages with proper async support (async-await, etc.),
> > we
> > > > can
> > > > > > skip sync API altogether.
> > > > > > It sounds pretty strange, as for me. Usually you cannot mix both
> > > > > functions
> > > > > > easily, it is called blue-red functions problem [1]
> > > > > > In python you definitely cannot do sync over async, for example
> > > > > > (principally can, but nobody do that because of mediocre
> > performance
> > > of
> > > > > > that solution). And many developers
> > > > > > still prefer writing code withou asyncio at all.
> > > > > >
> > > > > > Pavel, is it really true, that in .NET sync versions of libraries
> > and
> > > > > tools
> > > > > > are completely eliminated?
> > > > > >
> > > > > > [1] --
> > > > > >
> > > >
> > https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> > > > > >
> > > > > > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > >
> > > > > > > To put it another way:
> > > > > > > - true sync operation completes by itself
> > > > > > > - sync-over-async operation requires another thread to complete
> > > > > > >
> > > > > > > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn <
> > > ptupit...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > That's exactly what I have in mind.
> > > > > > > > Yes, we block the user thread, but then we should unblock it
> by
> > > > > > > completing
> > > > > > > > the future.
> > > > > > > > We can't complete the future from a Netty thread [1], so
> we'll
> > > need
> > > > > > some
> > > > > >

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Igor Sapego 
Well, fortunately we do not provide a C client if you don't consider ODBC
as one so we should not think about it. For C++ I believe we should use
standard std::future+std::promise for async, but still can provide sync
methods,
built on top of async methods. There is no continuation problem in C++ API
as
std::future do not provide continuation methods :)

Now, for the sync/async problem you've mentioned. Can not we solve it by
using different thread pools for completion and continuation of futures or
does
CompletableFuture interface not allowing for that?

Best Regards,
Igor


On Thu, Sep 9, 2021 at 1:15 PM Ivan Daschinsky  wrote:

> And what about C/C++?  There are so many options for them...
>
> чт, 9 сент. 2021 г. в 13:00, Pavel Tupitsyn :
>
> > Talked to Ivan in private, and came up with the following per-language
> > summary:
> >
> > * Java: sync and async
> > *. NET: only async
> > * Python: sync and async
> > * JavaScript: only async (sync is not possible)
> >
> > Thin clients in other languages are to be discussed.
> >
> > On Thu, Sep 9, 2021 at 11:49 AM Ivan Daschinsky 
> > wrote:
> >
> > > >> You can mix them easily.
> > > This is far from easily (you have already mentioned continuation
> > problem),
> > > but for i.e. in python it is absolutely not.
> > > For kotlin it is a little bit easier, but also not fluent and a little
> > bit
> > > ugly.
> > >
> > > чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn :
> > >
> > > > Ivan,
> > > >
> > > > > Pavel, is it really true, that in .NET sync versions of libraries
> and
> > > > tools
> > > > > are completely eliminated?
> > > >
> > > > Far from being eliminated, but there is a movement in this direction.
> > > > For example, HttpClient [1] only provides async variants for most of
> > the
> > > > methods.
> > > >
> > > > Exposing sync wrappers for async methods is not recommended [2].
> > > > There is a good explanation of potential threading issues there, and
> it
> > > can
> > > > be applied to Java too.
> > > >
> > > >
> > > > > you cannot mix both functions easily
> > > >
> > > > You can mix them easily.
> > > > C#: table.GetAsync(k).Result or
> > > table.GetAsync(k).GetAwaiter().GetResult()
> > > > (different exception handling)
> > > > Java: table.getAsync(k).get()
> > > >
> > > > The same thing we do in sync wrapper for async method is now in the
> > user
> > > > code.
> > > > Which is better for two reasons:
> > > > - users won't accidentally use sync API when async API should be used
> > > > - when they really have to use sync API, potentially dangerous
> behavior
> > > is
> > > > not hidden
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> > > > [2]
> > > >
> > > >
> > >
> >
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
> > > >
> > > > On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky  >
> > > > wrote:
> > > >
> > > > > >> 2. In languages with proper async support (async-await, etc.),
> we
> > > can
> > > > > skip sync API altogether.
> > > > > It sounds pretty strange, as for me. Usually you cannot mix both
> > > > functions
> > > > > easily, it is called blue-red functions problem [1]
> > > > > In python you definitely cannot do sync over async, for example
> > > > > (principally can, but nobody do that because of mediocre
> performance
> > of
> > > > > that solution). And many developers
> > > > > still prefer writing code withou asyncio at all.
> > > > >
> > > > > Pavel, is it really true, that in .NET sync versions of libraries
> and
> > > > tools
> > > > > are completely eliminated?
> > > > >
> > > > > [1] --
> > > > >
> > >
> https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> > > > >
> > > > > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn  >:
> > > > >
> > > > > > To put it another way:
> > > > > > - true sync operation completes by itself
> > > > > > - sync-over-async operation requires another thread to complete
> > > > > >
> > > > > > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Val,
> > > > > > >
> > > > > > > That's exactly what I have in mind.
> > > > > > > Yes, we block the user thread, but then we should unblock it by
> > > > > > completing
> > > > > > > the future.
> > > > > > > We can't complete the future from a Netty thread [1], so we'll
> > need
> > > > > some
> > > > > > > other thread from some executor.
> > > > > > > If there are no threads available (because they are blocked by
> > the
> > > > sync
> > > > > > > API above), the future won't complete => deadlock.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > > > > > >
> > > > > > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > > > > >

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Ivan Daschinsky 
And what about C/C++?  There are so many options for them...

чт, 9 сент. 2021 г. в 13:00, Pavel Tupitsyn :

> Talked to Ivan in private, and came up with the following per-language
> summary:
>
> * Java: sync and async
> *. NET: only async
> * Python: sync and async
> * JavaScript: only async (sync is not possible)
>
> Thin clients in other languages are to be discussed.
>
> On Thu, Sep 9, 2021 at 11:49 AM Ivan Daschinsky 
> wrote:
>
> > >> You can mix them easily.
> > This is far from easily (you have already mentioned continuation
> problem),
> > but for i.e. in python it is absolutely not.
> > For kotlin it is a little bit easier, but also not fluent and a little
> bit
> > ugly.
> >
> > чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn :
> >
> > > Ivan,
> > >
> > > > Pavel, is it really true, that in .NET sync versions of libraries and
> > > tools
> > > > are completely eliminated?
> > >
> > > Far from being eliminated, but there is a movement in this direction.
> > > For example, HttpClient [1] only provides async variants for most of
> the
> > > methods.
> > >
> > > Exposing sync wrappers for async methods is not recommended [2].
> > > There is a good explanation of potential threading issues there, and it
> > can
> > > be applied to Java too.
> > >
> > >
> > > > you cannot mix both functions easily
> > >
> > > You can mix them easily.
> > > C#: table.GetAsync(k).Result or
> > table.GetAsync(k).GetAwaiter().GetResult()
> > > (different exception handling)
> > > Java: table.getAsync(k).get()
> > >
> > > The same thing we do in sync wrapper for async method is now in the
> user
> > > code.
> > > Which is better for two reasons:
> > > - users won't accidentally use sync API when async API should be used
> > > - when they really have to use sync API, potentially dangerous behavior
> > is
> > > not hidden
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> > > [2]
> > >
> > >
> >
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
> > >
> > > On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky 
> > > wrote:
> > >
> > > > >> 2. In languages with proper async support (async-await, etc.), we
> > can
> > > > skip sync API altogether.
> > > > It sounds pretty strange, as for me. Usually you cannot mix both
> > > functions
> > > > easily, it is called blue-red functions problem [1]
> > > > In python you definitely cannot do sync over async, for example
> > > > (principally can, but nobody do that because of mediocre performance
> of
> > > > that solution). And many developers
> > > > still prefer writing code withou asyncio at all.
> > > >
> > > > Pavel, is it really true, that in .NET sync versions of libraries and
> > > tools
> > > > are completely eliminated?
> > > >
> > > > [1] --
> > > >
> > https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> > > >
> > > > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :
> > > >
> > > > > To put it another way:
> > > > > - true sync operation completes by itself
> > > > > - sync-over-async operation requires another thread to complete
> > > > >
> > > > > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn <
> ptupit...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > That's exactly what I have in mind.
> > > > > > Yes, we block the user thread, but then we should unblock it by
> > > > > completing
> > > > > > the future.
> > > > > > We can't complete the future from a Netty thread [1], so we'll
> need
> > > > some
> > > > > > other thread from some executor.
> > > > > > If there are no threads available (because they are blocked by
> the
> > > sync
> > > > > > API above), the future won't complete => deadlock.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > > > > >
> > > > > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com> wrote:
> > > > > >
> > > > > >> Pavel,
> > > > > >>
> > > > > >> I might be missing something - could you please elaborate a
> little
> > > > more?
> > > > > >>
> > > > > >> When I say "sync on top of async", I basically mean that (for
> > > example)
> > > > > >> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> > > > > >> understanding, it only blocks the user's thread.
> > > > > >>
> > > > > >> Am I wrong? Or you have a different implementation in mind?
> > > > > >>
> > > > > >> -Val
> > > > > >>
> > > > > >> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn <
> > > ptupit...@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Val,
> > > > > >> >
> > > > > >> > Agree with your points.
> > > > > >> >
> > > > > >> >
> > > > > >> > > async API should be primary
> > > > > >> >
> > > > > >> > It should be noted that all our APIs are inherently async,
> > > > > >> >

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Pavel Tupitsyn 
Talked to Ivan in private, and came up with the following per-language
summary:

* Java: sync and async
*. NET: only async
* Python: sync and async
* JavaScript: only async (sync is not possible)

Thin clients in other languages are to be discussed.

On Thu, Sep 9, 2021 at 11:49 AM Ivan Daschinsky  wrote:

> >> You can mix them easily.
> This is far from easily (you have already mentioned continuation problem),
> but for i.e. in python it is absolutely not.
> For kotlin it is a little bit easier, but also not fluent and a little bit
> ugly.
>
> чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn :
>
> > Ivan,
> >
> > > Pavel, is it really true, that in .NET sync versions of libraries and
> > tools
> > > are completely eliminated?
> >
> > Far from being eliminated, but there is a movement in this direction.
> > For example, HttpClient [1] only provides async variants for most of the
> > methods.
> >
> > Exposing sync wrappers for async methods is not recommended [2].
> > There is a good explanation of potential threading issues there, and it
> can
> > be applied to Java too.
> >
> >
> > > you cannot mix both functions easily
> >
> > You can mix them easily.
> > C#: table.GetAsync(k).Result or
> table.GetAsync(k).GetAwaiter().GetResult()
> > (different exception handling)
> > Java: table.getAsync(k).get()
> >
> > The same thing we do in sync wrapper for async method is now in the user
> > code.
> > Which is better for two reasons:
> > - users won't accidentally use sync API when async API should be used
> > - when they really have to use sync API, potentially dangerous behavior
> is
> > not hidden
> >
> >
> > [1]
> >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> > [2]
> >
> >
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
> >
> > On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky 
> > wrote:
> >
> > > >> 2. In languages with proper async support (async-await, etc.), we
> can
> > > skip sync API altogether.
> > > It sounds pretty strange, as for me. Usually you cannot mix both
> > functions
> > > easily, it is called blue-red functions problem [1]
> > > In python you definitely cannot do sync over async, for example
> > > (principally can, but nobody do that because of mediocre performance of
> > > that solution). And many developers
> > > still prefer writing code withou asyncio at all.
> > >
> > > Pavel, is it really true, that in .NET sync versions of libraries and
> > tools
> > > are completely eliminated?
> > >
> > > [1] --
> > >
> https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> > >
> > > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :
> > >
> > > > To put it another way:
> > > > - true sync operation completes by itself
> > > > - sync-over-async operation requires another thread to complete
> > > >
> > > > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > That's exactly what I have in mind.
> > > > > Yes, we block the user thread, but then we should unblock it by
> > > > completing
> > > > > the future.
> > > > > We can't complete the future from a Netty thread [1], so we'll need
> > > some
> > > > > other thread from some executor.
> > > > > If there are no threads available (because they are blocked by the
> > sync
> > > > > API above), the future won't complete => deadlock.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > > > >
> > > > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com> wrote:
> > > > >
> > > > >> Pavel,
> > > > >>
> > > > >> I might be missing something - could you please elaborate a little
> > > more?
> > > > >>
> > > > >> When I say "sync on top of async", I basically mean that (for
> > example)
> > > > >> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> > > > >> understanding, it only blocks the user's thread.
> > > > >>
> > > > >> Am I wrong? Or you have a different implementation in mind?
> > > > >>
> > > > >> -Val
> > > > >>
> > > > >> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn <
> > ptupit...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Val,
> > > > >> >
> > > > >> > Agree with your points.
> > > > >> >
> > > > >> >
> > > > >> > > async API should be primary
> > > > >> >
> > > > >> > It should be noted that all our APIs are inherently async,
> > > > >> > because thin client is implemented asynchronously.
> > > > >> >
> > > > >> >
> > > > >> > > with the sync version build on top
> > > > >> >
> > > > >> > We should document somehow that sync APIs are based on async
> ones,
> > > > >> > because this may be dangerous in some use cases.
> > > > >> >
> > > > >> > For example, as a user, I may have a thread pool of 4 threads
> for
> > > > >> > Ignite-related usage, that is also set as
> >

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Ivan Daschinsky 
>> You can mix them easily.
This is far from easily (you have already mentioned continuation problem),
but for i.e. in python it is absolutely not.
For kotlin it is a little bit easier, but also not fluent and a little bit
ugly.

чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn :

> Ivan,
>
> > Pavel, is it really true, that in .NET sync versions of libraries and
> tools
> > are completely eliminated?
>
> Far from being eliminated, but there is a movement in this direction.
> For example, HttpClient [1] only provides async variants for most of the
> methods.
>
> Exposing sync wrappers for async methods is not recommended [2].
> There is a good explanation of potential threading issues there, and it can
> be applied to Java too.
>
>
> > you cannot mix both functions easily
>
> You can mix them easily.
> C#: table.GetAsync(k).Result or table.GetAsync(k).GetAwaiter().GetResult()
> (different exception handling)
> Java: table.getAsync(k).get()
>
> The same thing we do in sync wrapper for async method is now in the user
> code.
> Which is better for two reasons:
> - users won't accidentally use sync API when async API should be used
> - when they really have to use sync API, potentially dangerous behavior is
> not hidden
>
>
> [1]
>
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> [2]
>
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
>
> On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky 
> wrote:
>
> > >> 2. In languages with proper async support (async-await, etc.), we can
> > skip sync API altogether.
> > It sounds pretty strange, as for me. Usually you cannot mix both
> functions
> > easily, it is called blue-red functions problem [1]
> > In python you definitely cannot do sync over async, for example
> > (principally can, but nobody do that because of mediocre performance of
> > that solution). And many developers
> > still prefer writing code withou asyncio at all.
> >
> > Pavel, is it really true, that in .NET sync versions of libraries and
> tools
> > are completely eliminated?
> >
> > [1] --
> > https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> >
> > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :
> >
> > > To put it another way:
> > > - true sync operation completes by itself
> > > - sync-over-async operation requires another thread to complete
> > >
> > > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Val,
> > > >
> > > > That's exactly what I have in mind.
> > > > Yes, we block the user thread, but then we should unblock it by
> > > completing
> > > > the future.
> > > > We can't complete the future from a Netty thread [1], so we'll need
> > some
> > > > other thread from some executor.
> > > > If there are no threads available (because they are blocked by the
> sync
> > > > API above), the future won't complete => deadlock.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > > >
> > > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > >> Pavel,
> > > >>
> > > >> I might be missing something - could you please elaborate a little
> > more?
> > > >>
> > > >> When I say "sync on top of async", I basically mean that (for
> example)
> > > >> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> > > >> understanding, it only blocks the user's thread.
> > > >>
> > > >> Am I wrong? Or you have a different implementation in mind?
> > > >>
> > > >> -Val
> > > >>
> > > >> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn <
> ptupit...@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Val,
> > > >> >
> > > >> > Agree with your points.
> > > >> >
> > > >> >
> > > >> > > async API should be primary
> > > >> >
> > > >> > It should be noted that all our APIs are inherently async,
> > > >> > because thin client is implemented asynchronously.
> > > >> >
> > > >> >
> > > >> > > with the sync version build on top
> > > >> >
> > > >> > We should document somehow that sync APIs are based on async ones,
> > > >> > because this may be dangerous in some use cases.
> > > >> >
> > > >> > For example, as a user, I may have a thread pool of 4 threads for
> > > >> > Ignite-related usage, that is also set as
> asyncContinuationExecutor
> > > [1].
> > > >> > Now if I run a lot of concurrent Ignite requests using sync API,
> > all 4
> > > >> > threads will end up blocked on CompletableFutures.
> > > >> > When one of the operations completes, we enqueue the completion to
> > > that
> > > >> > same thread pool, but all threads are blocked on sync APIs,
> > resulting
> > > >> in a
> > > >> > deadlock.
> > > >> >
> > > >> > This can be prevented by using a different
> > asyncContinuationExecutor,
> > > >> but
> > > >> > sync API users won't be usually aware of this.
> > > >> >
> > > >> >
> > > >> > [1]

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Pavel Tupitsyn 
Ivan,

> Pavel, is it really true, that in .NET sync versions of libraries and
tools
> are completely eliminated?

Far from being eliminated, but there is a movement in this direction.
For example, HttpClient [1] only provides async variants for most of the
methods.

Exposing sync wrappers for async methods is not recommended [2].
There is a good explanation of potential threading issues there, and it can
be applied to Java too.


> you cannot mix both functions easily

You can mix them easily.
C#: table.GetAsync(k).Result or table.GetAsync(k).GetAwaiter().GetResult()
(different exception handling)
Java: table.getAsync(k).get()

The same thing we do in sync wrapper for async method is now in the user
code.
Which is better for two reasons:
- users won't accidentally use sync API when async API should be used
- when they really have to use sync API, potentially dangerous behavior is
not hidden


[1]
https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
[2]
https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/

On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky  wrote:

> >> 2. In languages with proper async support (async-await, etc.), we can
> skip sync API altogether.
> It sounds pretty strange, as for me. Usually you cannot mix both functions
> easily, it is called blue-red functions problem [1]
> In python you definitely cannot do sync over async, for example
> (principally can, but nobody do that because of mediocre performance of
> that solution). And many developers
> still prefer writing code withou asyncio at all.
>
> Pavel, is it really true, that in .NET sync versions of libraries and tools
> are completely eliminated?
>
> [1] --
> https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
>
> ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :
>
> > To put it another way:
> > - true sync operation completes by itself
> > - sync-over-async operation requires another thread to complete
> >
> > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn 
> > wrote:
> >
> > > Val,
> > >
> > > That's exactly what I have in mind.
> > > Yes, we block the user thread, but then we should unblock it by
> > completing
> > > the future.
> > > We can't complete the future from a Netty thread [1], so we'll need
> some
> > > other thread from some executor.
> > > If there are no threads available (because they are blocked by the sync
> > > API above), the future won't complete => deadlock.
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > >
> > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > >> Pavel,
> > >>
> > >> I might be missing something - could you please elaborate a little
> more?
> > >>
> > >> When I say "sync on top of async", I basically mean that (for example)
> > >> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> > >> understanding, it only blocks the user's thread.
> > >>
> > >> Am I wrong? Or you have a different implementation in mind?
> > >>
> > >> -Val
> > >>
> > >> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
> > >> wrote:
> > >>
> > >> > Val,
> > >> >
> > >> > Agree with your points.
> > >> >
> > >> >
> > >> > > async API should be primary
> > >> >
> > >> > It should be noted that all our APIs are inherently async,
> > >> > because thin client is implemented asynchronously.
> > >> >
> > >> >
> > >> > > with the sync version build on top
> > >> >
> > >> > We should document somehow that sync APIs are based on async ones,
> > >> > because this may be dangerous in some use cases.
> > >> >
> > >> > For example, as a user, I may have a thread pool of 4 threads for
> > >> > Ignite-related usage, that is also set as asyncContinuationExecutor
> > [1].
> > >> > Now if I run a lot of concurrent Ignite requests using sync API,
> all 4
> > >> > threads will end up blocked on CompletableFutures.
> > >> > When one of the operations completes, we enqueue the completion to
> > that
> > >> > same thread pool, but all threads are blocked on sync APIs,
> resulting
> > >> in a
> > >> > deadlock.
> > >> >
> > >> > This can be prevented by using a different
> asyncContinuationExecutor,
> > >> but
> > >> > sync API users won't be usually aware of this.
> > >> >
> > >> >
> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-15359
> > >> >
> > >> > On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
> > >> > courtney.robin...@hypi.io>
> > >> > wrote:
> > >> >
> > >> > > Hi Val,
> > >> > >
> > >> > > I'd highly support an async first API based on CompletionStage
> > >> > > <
> > >> > >
> > >> >
> > >>
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> > >> > > >
> > >> > > or
> > >> > > its subtypes like CompletableFuture.
> > >> > > In Ignite 2 we've written a wrapper library around IgniteFuture to
> > >> >

Re: Sync vs async APIs in Ignite 3

2021-09-09 Thread Ivan Daschinsky 
>> 2. In languages with proper async support (async-await, etc.), we can
skip sync API altogether.
It sounds pretty strange, as for me. Usually you cannot mix both functions
easily, it is called blue-red functions problem [1]
In python you definitely cannot do sync over async, for example
(principally can, but nobody do that because of mediocre performance of
that solution). And many developers
still prefer writing code withou asyncio at all.

Pavel, is it really true, that in .NET sync versions of libraries and tools
are completely eliminated?

[1] --
https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d

ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :

> To put it another way:
> - true sync operation completes by itself
> - sync-over-async operation requires another thread to complete
>
> On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn 
> wrote:
>
> > Val,
> >
> > That's exactly what I have in mind.
> > Yes, we block the user thread, but then we should unblock it by
> completing
> > the future.
> > We can't complete the future from a Netty thread [1], so we'll need some
> > other thread from some executor.
> > If there are no threads available (because they are blocked by the sync
> > API above), the future won't complete => deadlock.
> >
> > [1]
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> >
> > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> >> Pavel,
> >>
> >> I might be missing something - could you please elaborate a little more?
> >>
> >> When I say "sync on top of async", I basically mean that (for example)
> >> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> >> understanding, it only blocks the user's thread.
> >>
> >> Am I wrong? Or you have a different implementation in mind?
> >>
> >> -Val
> >>
> >> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
> >> wrote:
> >>
> >> > Val,
> >> >
> >> > Agree with your points.
> >> >
> >> >
> >> > > async API should be primary
> >> >
> >> > It should be noted that all our APIs are inherently async,
> >> > because thin client is implemented asynchronously.
> >> >
> >> >
> >> > > with the sync version build on top
> >> >
> >> > We should document somehow that sync APIs are based on async ones,
> >> > because this may be dangerous in some use cases.
> >> >
> >> > For example, as a user, I may have a thread pool of 4 threads for
> >> > Ignite-related usage, that is also set as asyncContinuationExecutor
> [1].
> >> > Now if I run a lot of concurrent Ignite requests using sync API, all 4
> >> > threads will end up blocked on CompletableFutures.
> >> > When one of the operations completes, we enqueue the completion to
> that
> >> > same thread pool, but all threads are blocked on sync APIs, resulting
> >> in a
> >> > deadlock.
> >> >
> >> > This can be prevented by using a different asyncContinuationExecutor,
> >> but
> >> > sync API users won't be usually aware of this.
> >> >
> >> >
> >> > [1] https://issues.apache.org/jira/browse/IGNITE-15359
> >> >
> >> > On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
> >> > courtney.robin...@hypi.io>
> >> > wrote:
> >> >
> >> > > Hi Val,
> >> > >
> >> > > I'd highly support an async first API based on CompletionStage
> >> > > <
> >> > >
> >> >
> >>
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> >> > > >
> >> > > or
> >> > > its subtypes like CompletableFuture.
> >> > > In Ignite 2 we've written a wrapper library around IgniteFuture to
> >> > provide
> >> > > CompletionStage instead because many of the newer libs we use
> support
> >> > this.
> >> > > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper
> >> that
> >> > we
> >> > > wrote to get what you're suggesting here.
> >> > >
> >> > > Regards,
> >> > > Courtney Robinson
> >> > > Founder and CEO, Hypi
> >> > > Tel: ++44 208 123 2413 (GMT+0) 
> >> > >
> >> > > 
> >> > > https://hypi.io
> >> > >
> >> > >
> >> > > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> >> > > valentin.kuliche...@gmail.com> wrote:
> >> > >
> >> > > > Igniters,
> >> > > >
> >> > > > I would like to gather some opinions on whether we want to focus
> on
> >> > sync
> >> > > vs
> >> > > > async APIs in Ignite 3.
> >> > > >
> >> > > > Here are some initial considerations that I have:
> >> > > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but
> >> they
> >> > use
> >> > > > non-standard IgniteFuture and provide counterintuitive guarantees.
> >> In
> >> > my
> >> > > > experience, they significantly lack usability, and because of that
> >> are
> >> > > > rarely used.
> >> > > > 2. In general, however, async execution becomes more and more
> >> > prominent.
> >> > > > Something we can't ignore if we want to create a modern framework.
> >> > > > 3. Still, async support in Java is very limited (especially if
> >> compared

Re: Sync vs async APIs in Ignite 3

2021-09-08 Thread Pavel Tupitsyn 
To put it another way:
- true sync operation completes by itself
- sync-over-async operation requires another thread to complete

On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn  wrote:

> Val,
>
> That's exactly what I have in mind.
> Yes, we block the user thread, but then we should unblock it by completing
> the future.
> We can't complete the future from a Netty thread [1], so we'll need some
> other thread from some executor.
> If there are no threads available (because they are blocked by the sync
> API above), the future won't complete => deadlock.
>
> [1]
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
>
> On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Pavel,
>>
>> I might be missing something - could you please elaborate a little more?
>>
>> When I say "sync on top of async", I basically mean that (for example)
>> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
>> understanding, it only blocks the user's thread.
>>
>> Am I wrong? Or you have a different implementation in mind?
>>
>> -Val
>>
>> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
>> wrote:
>>
>> > Val,
>> >
>> > Agree with your points.
>> >
>> >
>> > > async API should be primary
>> >
>> > It should be noted that all our APIs are inherently async,
>> > because thin client is implemented asynchronously.
>> >
>> >
>> > > with the sync version build on top
>> >
>> > We should document somehow that sync APIs are based on async ones,
>> > because this may be dangerous in some use cases.
>> >
>> > For example, as a user, I may have a thread pool of 4 threads for
>> > Ignite-related usage, that is also set as asyncContinuationExecutor [1].
>> > Now if I run a lot of concurrent Ignite requests using sync API, all 4
>> > threads will end up blocked on CompletableFutures.
>> > When one of the operations completes, we enqueue the completion to that
>> > same thread pool, but all threads are blocked on sync APIs, resulting
>> in a
>> > deadlock.
>> >
>> > This can be prevented by using a different asyncContinuationExecutor,
>> but
>> > sync API users won't be usually aware of this.
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-15359
>> >
>> > On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
>> > courtney.robin...@hypi.io>
>> > wrote:
>> >
>> > > Hi Val,
>> > >
>> > > I'd highly support an async first API based on CompletionStage
>> > > <
>> > >
>> >
>> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
>> > > >
>> > > or
>> > > its subtypes like CompletableFuture.
>> > > In Ignite 2 we've written a wrapper library around IgniteFuture to
>> > provide
>> > > CompletionStage instead because many of the newer libs we use support
>> > this.
>> > > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper
>> that
>> > we
>> > > wrote to get what you're suggesting here.
>> > >
>> > > Regards,
>> > > Courtney Robinson
>> > > Founder and CEO, Hypi
>> > > Tel: ++44 208 123 2413 (GMT+0) 
>> > >
>> > > 
>> > > https://hypi.io
>> > >
>> > >
>> > > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
>> > > valentin.kuliche...@gmail.com> wrote:
>> > >
>> > > > Igniters,
>> > > >
>> > > > I would like to gather some opinions on whether we want to focus on
>> > sync
>> > > vs
>> > > > async APIs in Ignite 3.
>> > > >
>> > > > Here are some initial considerations that I have:
>> > > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but
>> they
>> > use
>> > > > non-standard IgniteFuture and provide counterintuitive guarantees.
>> In
>> > my
>> > > > experience, they significantly lack usability, and because of that
>> are
>> > > > rarely used.
>> > > > 2. In general, however, async execution becomes more and more
>> > prominent.
>> > > > Something we can't ignore if we want to create a modern framework.
>> > > > 3. Still, async support in Java is very limited (especially if
>> compared
>> > > to
>> > > > other languages, like C# for example).
>> > > >
>> > > > My current position is the following (happy to discuss):
>> > > > 1. We should pay more attention to async APIs. As a general rule,
>> async
>> > > API
>> > > > should be primary, with the sync version build on top.
>> > > > 2. In languages with proper async support (async-await, etc.), we
>> can
>> > > skip
>> > > > sync API altogether. As an example of this, you can look at the
>> first
>> > > > version of the .NET client [1]. It exposes only async methods, and
>> it
>> > > > doesn't look like sync counterparts are really needed.
>> > > > 3. In Java (as well as other languages where applicable), we will
>> add
>> > > sync
>> > > > APIs that simply delegate to async APIs. This will help users to
>> avoid
>> > > > CompletableFuture if they don't want to use it.
>> > > >
>> > > > [1] https://github.com/apache/ignite-3/pull/306
>> > > >
>> > > > Please share

Re: Sync vs async APIs in Ignite 3

2021-09-08 Thread Pavel Tupitsyn 
Val,

That's exactly what I have in mind.
Yes, we block the user thread, but then we should unblock it by completing
the future.
We can't complete the future from a Netty thread [1], so we'll need some
other thread from some executor.
If there are no threads available (because they are blocked by the sync API
above), the future won't complete => deadlock.

[1]
https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E

On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> I might be missing something - could you please elaborate a little more?
>
> When I say "sync on top of async", I basically mean that (for example)
> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> understanding, it only blocks the user's thread.
>
> Am I wrong? Or you have a different implementation in mind?
>
> -Val
>
> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
> wrote:
>
> > Val,
> >
> > Agree with your points.
> >
> >
> > > async API should be primary
> >
> > It should be noted that all our APIs are inherently async,
> > because thin client is implemented asynchronously.
> >
> >
> > > with the sync version build on top
> >
> > We should document somehow that sync APIs are based on async ones,
> > because this may be dangerous in some use cases.
> >
> > For example, as a user, I may have a thread pool of 4 threads for
> > Ignite-related usage, that is also set as asyncContinuationExecutor [1].
> > Now if I run a lot of concurrent Ignite requests using sync API, all 4
> > threads will end up blocked on CompletableFutures.
> > When one of the operations completes, we enqueue the completion to that
> > same thread pool, but all threads are blocked on sync APIs, resulting in
> a
> > deadlock.
> >
> > This can be prevented by using a different asyncContinuationExecutor, but
> > sync API users won't be usually aware of this.
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15359
> >
> > On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
> > courtney.robin...@hypi.io>
> > wrote:
> >
> > > Hi Val,
> > >
> > > I'd highly support an async first API based on CompletionStage
> > > <
> > >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> > > >
> > > or
> > > its subtypes like CompletableFuture.
> > > In Ignite 2 we've written a wrapper library around IgniteFuture to
> > provide
> > > CompletionStage instead because many of the newer libs we use support
> > this.
> > > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper
> that
> > we
> > > wrote to get what you're suggesting here.
> > >
> > > Regards,
> > > Courtney Robinson
> > > Founder and CEO, Hypi
> > > Tel: ++44 208 123 2413 (GMT+0) 
> > >
> > > 
> > > https://hypi.io
> > >
> > >
> > > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Igniters,
> > > >
> > > > I would like to gather some opinions on whether we want to focus on
> > sync
> > > vs
> > > > async APIs in Ignite 3.
> > > >
> > > > Here are some initial considerations that I have:
> > > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they
> > use
> > > > non-standard IgniteFuture and provide counterintuitive guarantees. In
> > my
> > > > experience, they significantly lack usability, and because of that
> are
> > > > rarely used.
> > > > 2. In general, however, async execution becomes more and more
> > prominent.
> > > > Something we can't ignore if we want to create a modern framework.
> > > > 3. Still, async support in Java is very limited (especially if
> compared
> > > to
> > > > other languages, like C# for example).
> > > >
> > > > My current position is the following (happy to discuss):
> > > > 1. We should pay more attention to async APIs. As a general rule,
> async
> > > API
> > > > should be primary, with the sync version build on top.
> > > > 2. In languages with proper async support (async-await, etc.), we can
> > > skip
> > > > sync API altogether. As an example of this, you can look at the first
> > > > version of the .NET client [1]. It exposes only async methods, and it
> > > > doesn't look like sync counterparts are really needed.
> > > > 3. In Java (as well as other languages where applicable), we will add
> > > sync
> > > > APIs that simply delegate to async APIs. This will help users to
> avoid
> > > > CompletableFuture if they don't want to use it.
> > > >
> > > > [1] https://github.com/apache/ignite-3/pull/306
> > > >
> > > > Please share your thoughts.
> > > >
> > > > -Val
> > > >
> > >
> >
>

Re: Sync vs async APIs in Ignite 3

2021-09-08 Thread Valentin Kulichenko 
Pavel,

I might be missing something - could you please elaborate a little more?

When I say "sync on top of async", I basically mean that (for example)
'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
understanding, it only blocks the user's thread.

Am I wrong? Or you have a different implementation in mind?

-Val

On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn  wrote:

> Val,
>
> Agree with your points.
>
>
> > async API should be primary
>
> It should be noted that all our APIs are inherently async,
> because thin client is implemented asynchronously.
>
>
> > with the sync version build on top
>
> We should document somehow that sync APIs are based on async ones,
> because this may be dangerous in some use cases.
>
> For example, as a user, I may have a thread pool of 4 threads for
> Ignite-related usage, that is also set as asyncContinuationExecutor [1].
> Now if I run a lot of concurrent Ignite requests using sync API, all 4
> threads will end up blocked on CompletableFutures.
> When one of the operations completes, we enqueue the completion to that
> same thread pool, but all threads are blocked on sync APIs, resulting in a
> deadlock.
>
> This can be prevented by using a different asyncContinuationExecutor, but
> sync API users won't be usually aware of this.
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-15359
>
> On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
> courtney.robin...@hypi.io>
> wrote:
>
> > Hi Val,
> >
> > I'd highly support an async first API based on CompletionStage
> > <
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> > >
> > or
> > its subtypes like CompletableFuture.
> > In Ignite 2 we've written a wrapper library around IgniteFuture to
> provide
> > CompletionStage instead because many of the newer libs we use support
> this.
> > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper that
> we
> > wrote to get what you're suggesting here.
> >
> > Regards,
> > Courtney Robinson
> > Founder and CEO, Hypi
> > Tel: ++44 208 123 2413 (GMT+0) 
> >
> > 
> > https://hypi.io
> >
> >
> > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Igniters,
> > >
> > > I would like to gather some opinions on whether we want to focus on
> sync
> > vs
> > > async APIs in Ignite 3.
> > >
> > > Here are some initial considerations that I have:
> > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they
> use
> > > non-standard IgniteFuture and provide counterintuitive guarantees. In
> my
> > > experience, they significantly lack usability, and because of that are
> > > rarely used.
> > > 2. In general, however, async execution becomes more and more
> prominent.
> > > Something we can't ignore if we want to create a modern framework.
> > > 3. Still, async support in Java is very limited (especially if compared
> > to
> > > other languages, like C# for example).
> > >
> > > My current position is the following (happy to discuss):
> > > 1. We should pay more attention to async APIs. As a general rule, async
> > API
> > > should be primary, with the sync version build on top.
> > > 2. In languages with proper async support (async-await, etc.), we can
> > skip
> > > sync API altogether. As an example of this, you can look at the first
> > > version of the .NET client [1]. It exposes only async methods, and it
> > > doesn't look like sync counterparts are really needed.
> > > 3. In Java (as well as other languages where applicable), we will add
> > sync
> > > APIs that simply delegate to async APIs. This will help users to avoid
> > > CompletableFuture if they don't want to use it.
> > >
> > > [1] https://github.com/apache/ignite-3/pull/306
> > >
> > > Please share your thoughts.
> > >
> > > -Val
> > >
> >
>

Re: Sync vs async APIs in Ignite 3

2021-09-08 Thread Pavel Tupitsyn 
Val,

Agree with your points.


> async API should be primary

It should be noted that all our APIs are inherently async,
because thin client is implemented asynchronously.


> with the sync version build on top

We should document somehow that sync APIs are based on async ones,
because this may be dangerous in some use cases.

For example, as a user, I may have a thread pool of 4 threads for
Ignite-related usage, that is also set as asyncContinuationExecutor [1].
Now if I run a lot of concurrent Ignite requests using sync API, all 4
threads will end up blocked on CompletableFutures.
When one of the operations completes, we enqueue the completion to that
same thread pool, but all threads are blocked on sync APIs, resulting in a
deadlock.

This can be prevented by using a different asyncContinuationExecutor, but
sync API users won't be usually aware of this.


[1] https://issues.apache.org/jira/browse/IGNITE-15359

On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson 
wrote:

> Hi Val,
>
> I'd highly support an async first API based on CompletionStage
> <
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> >
> or
> its subtypes like CompletableFuture.
> In Ignite 2 we've written a wrapper library around IgniteFuture to provide
> CompletionStage instead because many of the newer libs we use support this.
> If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper that we
> wrote to get what you're suggesting here.
>
> Regards,
> Courtney Robinson
> Founder and CEO, Hypi
> Tel: ++44 208 123 2413 (GMT+0) 
>
> 
> https://hypi.io
>
>
> On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Igniters,
> >
> > I would like to gather some opinions on whether we want to focus on sync
> vs
> > async APIs in Ignite 3.
> >
> > Here are some initial considerations that I have:
> > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they use
> > non-standard IgniteFuture and provide counterintuitive guarantees. In my
> > experience, they significantly lack usability, and because of that are
> > rarely used.
> > 2. In general, however, async execution becomes more and more prominent.
> > Something we can't ignore if we want to create a modern framework.
> > 3. Still, async support in Java is very limited (especially if compared
> to
> > other languages, like C# for example).
> >
> > My current position is the following (happy to discuss):
> > 1. We should pay more attention to async APIs. As a general rule, async
> API
> > should be primary, with the sync version build on top.
> > 2. In languages with proper async support (async-await, etc.), we can
> skip
> > sync API altogether. As an example of this, you can look at the first
> > version of the .NET client [1]. It exposes only async methods, and it
> > doesn't look like sync counterparts are really needed.
> > 3. In Java (as well as other languages where applicable), we will add
> sync
> > APIs that simply delegate to async APIs. This will help users to avoid
> > CompletableFuture if they don't want to use it.
> >
> > [1] https://github.com/apache/ignite-3/pull/306
> >
> > Please share your thoughts.
> >
> > -Val
> >
>

Re: Sync vs async APIs in Ignite 3

2021-09-08 Thread Courtney Robinson 
Hi Val,

I'd highly support an async first API based on CompletionStage

or
its subtypes like CompletableFuture.
In Ignite 2 we've written a wrapper library around IgniteFuture to provide
CompletionStage instead because many of the newer libs we use support this.
If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper that we
wrote to get what you're suggesting here.

Regards,
Courtney Robinson
Founder and CEO, Hypi
Tel: ++44 208 123 2413 (GMT+0) 


https://hypi.io


On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Igniters,
>
> I would like to gather some opinions on whether we want to focus on sync vs
> async APIs in Ignite 3.
>
> Here are some initial considerations that I have:
> 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they use
> non-standard IgniteFuture and provide counterintuitive guarantees. In my
> experience, they significantly lack usability, and because of that are
> rarely used.
> 2. In general, however, async execution becomes more and more prominent.
> Something we can't ignore if we want to create a modern framework.
> 3. Still, async support in Java is very limited (especially if compared to
> other languages, like C# for example).
>
> My current position is the following (happy to discuss):
> 1. We should pay more attention to async APIs. As a general rule, async API
> should be primary, with the sync version build on top.
> 2. In languages with proper async support (async-await, etc.), we can skip
> sync API altogether. As an example of this, you can look at the first
> version of the .NET client [1]. It exposes only async methods, and it
> doesn't look like sync counterparts are really needed.
> 3. In Java (as well as other languages where applicable), we will add sync
> APIs that simply delegate to async APIs. This will help users to avoid
> CompletableFuture if they don't want to use it.
>
> [1] https://github.com/apache/ignite-3/pull/306
>
> Please share your thoughts.
>
> -Val
>