Re: Assign an issue to myself

[Pavel Tupitsyn]

Hi Raymond,

I've added you to the Contributors group in JIRA, now you should be able to
assign issues, change their status, etc.

On Mon, Sep 27, 2021 at 12:25 AM Raymond Wilson 
wrote:

> Hi,
>
> I have created a Jira ticket I would like to assign to myself:
> https://issues.apache.org/jira/browse/IGNITE-15602
>
> Thanks,
> Raymond.
>
>
> --
> 
> Raymond Wilson
> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> 11 Birmingham Drive | Christchurch, New Zealand
> raymond_wil...@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Pavel Tupitsyn]

+1

On Fri, Sep 24, 2021 at 8:04 PM Maxim Muzafarov  wrote:

> Hello Nikita,
>
> +1 if you will lead this release.
>
>
> I'd suggest including into the release scope this one issue too:
>
> Snapshot restore must support restoring with indexes rebuild
> https://issues.apache.org/jira/browse/IGNITE-14744
>
> Hopefully, it will be completed by the end of October.
>
> On Fri, 24 Sept 2021 at 19:48, Nikita Amelchev 
> wrote:
> >
> > Dear Ignite Community!
> >
> > I suggest starting Apache Ignite 2.12 release activities.
> >
> > For now, we've accumulated a hundred resolved issues with new features
> > and bug fixes which are waiting for their release date. For example,
> >
> > CDC Application
> > Index Query API
> > Utility for analyzing indexes
> > Java compute tasks for C++
> > SQL commands statistics
> > etc.
> >
> > I want to propose myself to be the release manager of the planning
> release.
> >
> > I propose the following timeline:
> >
> > Scope Freeze: October 15, 2021
> > Code Freeze: October 29, 2021
> > Voting Date: November 15, 2021
> > Release Date: November 22, 2021
> >
> > WDYT?
> >
> > --
> > Best wishes,
> > Amelchev Nikita
>

Re: [RESULT][VOTE] Release Apache Ignite 2.11.0 RC2

[Pavel Tupitsyn]

Maxim,

I'll prepare a dedicated blog post about Ignite.NET changes later this week.
Thanks for driving the release!

On Mon, Sep 20, 2021 at 11:20 PM Maxim Muzafarov  wrote:

> Denis,
>
> The post is added:
> https://blogs.apache.org/ignite/entry/apache-ignite-2-11-stabilization
>
> I've fixed the image according to your suggestions.
>
> On Mon, 20 Sept 2021 at 21:35, Denis Magda  wrote:
> >
> > Maxim, thanks for driving the release and preparing the announcement
> > article!
> >
> > A few questions:
> >
> >- Should "dev" be replaced with "2.11" on the picture of the cellular
> >clusters deployment section?
> >- Will the article be published on our blogs.apache.org page?
> >
> > -
> > Denis
> >
> >
> > On Sat, Sep 18, 2021 at 9:36 PM Maxim Muzafarov 
> wrote:
> >
> > > Pavel,
> > >
> > >
> > > I've prepared a draft blog post [1] for the 2.11 release announcement
> > > message, however, this post doesn't include changes related to the
> > > .NET part of the Apache Ignite. Will you prepare a dedicated post
> > > related to the .NET changes or I should include them in this one?
> > >
> > > Folks,
> > > all the comments are welcome.
> > >
> > >
> > > [1]
> > >
> https://github.com/Mmuzaf/mmuzaf.github.io/blob/main/_post/Release_Newsletter_211.md
> > >
> > >
> > > On Thu, 16 Sept 2021 at 17:21, Maxim Muzafarov 
> wrote:
> > > >
> > > > Hello, Igniters!
> > > >
> > > > Apache Ignite 2.11.0 release (RC2) has been accepted.
> > > >
> > > > 5 - "+1" votes received.
> > > > 1 - "+0,5" vote received.
> > > >
> > > > Here are the votes received:
> > > >
> > > >  - Pavel Tupitsyn (binding)
> > > >  - Alex Plehanov (binding)
> > > >  - Nikolay Izhikov (binding)
> > > >  - Ilya Kasnacheev
> > > >  - Ivan Daschinsky
> > > >  - Zhenya Stanilovsky
> > > >
> > > >
> > > > Here is the link to the voting thread -
> > > >
> > >
> https://lists.apache.org/thread.html/rab233aa7d737b84b678fb74c81a867e3aad270471a2aac85a4d35cb8%40%3Cdev.ignite.apache.org%3E
> > > >
> > > > Thank you!
> > >
>

Re: [DISCUSSION] Remove VS project from C++

[Pavel Tupitsyn]

Ivan, thanks for the answers! I retract my "-1". No objections.

On Wed, Sep 15, 2021 at 12:58 PM Ivan Daschinsky 
wrote:

> >> 1. What percentage of our users rely on existing VS projects?
> It is impossible to answer, but I suppose not so much. And, again, these
> projects needs some tweaks in order to work properly in VS 2015+
> Moreover, you cannot use VS as is, you should tweak them or download
> dependencies in specific location.
>
> >> How much time (in minutes) does it take to switch from using existing VS
> projects to CMake-generated ones?
> If you use VS 2017 and later -- just a second, they already support CMake
> out of box
> https://devblogs.microsoft.com/cppblog/cmake-support-in-visual-studio/
> If you use VS2015 and earlielr -- just download cmake and use bundled gui
> app and generate projects in few clicks.
>
> And don't forget about vcpkg [1]. Even microsoft goes towards cmake and
> make even package manager for it.
> And it works great, I've tried with Ignite C++.
>
> [1] -- https://github.com/microsoft/vcpkg
>
> ср, 15 сент. 2021 г. в 12:46, Pavel Tupitsyn :
>
> > Ivan,
> >
> > Ok, I've got your point.
> > What's your assessment on the following:
> >
> > 1. What percentage of our users rely on existing VS projects?
> > 2. How much time (in minutes) does it take to switch from using existing
> VS
> > projects to CMake-generated ones?
> >
> >
> >
> > On Wed, Sep 15, 2021 at 12:37 PM Ivan Daschinsky 
> > wrote:
> >
> > > >> Currently user
> > > Sorry typo, I meant developer.
> > >
> > > ср, 15 сент. 2021 г. в 12:35, Ivan Daschinsky :
> > >
> > > > > How?
> > > > Currently user must add sources in 2 different places. One of this
> > places
> > > > is not specified and not intended to be edit outside VC.
> > > > CMake can generate VS projects easily and without any problem. I've
> > done
> > > > this even in 2008 when I was C++/Qt developer.
> > > >
> > > > >> 1. Get all files in the directory
> > > > >> 2. Filter by TestSuite suffix
> > > > >> 3. Check if all of them are present in VS files
> > > > First of all, BOOST_TEST is not NUnit :) Secondly, we can do that,
> but
> > it
> > > > is not so easy as it is in .NET.
> > > > Yes, we can use boost libraries in tests, but why we should do it?
> Why
> > we
> > > > should invest our time in this activity?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ср, 15 сент. 2021 г. в 12:30, Pavel Tupitsyn :
> > > >
> > > >> > It makes development much more easier.
> > > >>
> > > >> How?
> > > >>
> > > >> > I can hardly imagine how it can be done
> > > >>
> > > >> 1. Get all files in the directory
> > > >> 2. Filter by TestSuite suffix
> > > >> 3. Check if all of them are present in VS files
> > > >> Am I missing something? We have checks like this for Ignite.NET [1]
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core.Tests/ProjectFilesTest.cs
> > > >>
> > > >>
> > > >> On Wed, Sep 15, 2021 at 12:22 PM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > >> This may become an obstacle for some of the users and I'm not
> > sure
> > > >> how
> > > >> > it improves anything.
> > > >> > Please specify more correctly. What is an obstacle? Current VS
> > > projects
> > > >> > (odbc) cannot be build on VC 2015+
> > > >> > without modification. CMake is an industry standard now.
> > > >> > >> I'm not sure how it improves anything.
> > > >> > It makes development much more easier.
> > > >> >
> > > >> > >> We can add an automatic check for this (in form of a test).
> > > >> > I can hardly imagine how it can be done. And even if it is
> possible,
> > > >> this
> > > >> > is a sisyphus job.
> > > >> >
> > > >> >
> > > >> > ср, 15 сент. 2021 г. в 12:11, Petr Ivanov :

Re: [DISCUSSION] Remove VS project from C++

[Pavel Tupitsyn]

Ivan,

Ok, I've got your point.
What's your assessment on the following:

1. What percentage of our users rely on existing VS projects?
2. How much time (in minutes) does it take to switch from using existing VS
projects to CMake-generated ones?



On Wed, Sep 15, 2021 at 12:37 PM Ivan Daschinsky 
wrote:

> >> Currently user
> Sorry typo, I meant developer.
>
> ср, 15 сент. 2021 г. в 12:35, Ivan Daschinsky :
>
> > > How?
> > Currently user must add sources in 2 different places. One of this places
> > is not specified and not intended to be edit outside VC.
> > CMake can generate VS projects easily and without any problem. I've done
> > this even in 2008 when I was C++/Qt developer.
> >
> > >> 1. Get all files in the directory
> > >> 2. Filter by TestSuite suffix
> > >> 3. Check if all of them are present in VS files
> > First of all, BOOST_TEST is not NUnit :) Secondly, we can do that, but it
> > is not so easy as it is in .NET.
> > Yes, we can use boost libraries in tests, but why we should do it? Why we
> > should invest our time in this activity?
> >
> >
> >
> >
> >
> > ср, 15 сент. 2021 г. в 12:30, Pavel Tupitsyn :
> >
> >> > It makes development much more easier.
> >>
> >> How?
> >>
> >> > I can hardly imagine how it can be done
> >>
> >> 1. Get all files in the directory
> >> 2. Filter by TestSuite suffix
> >> 3. Check if all of them are present in VS files
> >> Am I missing something? We have checks like this for Ignite.NET [1]
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core.Tests/ProjectFilesTest.cs
> >>
> >>
> >> On Wed, Sep 15, 2021 at 12:22 PM Ivan Daschinsky 
> >> wrote:
> >>
> >> > >> This may become an obstacle for some of the users and I'm not sure
> >> how
> >> > it improves anything.
> >> > Please specify more correctly. What is an obstacle? Current VS
> projects
> >> > (odbc) cannot be build on VC 2015+
> >> > without modification. CMake is an industry standard now.
> >> > >> I'm not sure how it improves anything.
> >> > It makes development much more easier.
> >> >
> >> > >> We can add an automatic check for this (in form of a test).
> >> > I can hardly imagine how it can be done. And even if it is possible,
> >> this
> >> > is a sisyphus job.
> >> >
> >> >
> >> > ср, 15 сент. 2021 г. в 12:11, Petr Ivanov :
> >> >
> >> > > +1
> >> > >
> >> > > Let's keep the project clean and on the verge of preferable tech
> >> stack.
> >> > >
> >> > >
> >> > > > On 15 Sep 2021, at 12:02, Ivan Pavlukhin 
> >> wrote:
> >> > > >
> >> > > > +1 for removing VS project
> >> > > >
> >> > > > 2021-09-15 12:01 GMT+03:00, Nikolay Izhikov  >:
> >> > > >> +1
> >> > > >>
> >> > > >>> 15 сент. 2021 г., в 11:57, Pavel Tupitsyn  >
> >> > > >>> написал(а):
> >> > > >>>
> >> > > >>> -1
> >> > > >>>
> >> > > >>> This may become an obstacle for some of the users and I'm not
> sure
> >> > how
> >> > > it
> >> > > >>> improves anything.
> >> > > >>>
> >> > > >>>> 3. Sometimes even maintainers forget to add test sources to VS
> >> > > projects
> >> > > >>> [1]
> >> > > >>> We can add an automatic check for this (in form of a test).
> >> > > >>>
> >> > > >>> On Wed, Sep 15, 2021 at 10:28 AM Zhenya Stanilovsky
> >> > > >>>  wrote:
> >> > > >>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> completely support !
> >> > > >>>>
> >> > > >>>>> Igniters!
> >> > > >>>>>
> >> > > >>>>> Currently we have CMake build system, that works on Windows,
> >> Linux
> >> > > and
> >> > > >>>>> MacOs flawlessly
> >> > > >>>>>
> >> > > >>>>> 1. CMake is supported natively in VS 2019
> >> > > >>>>> 2. CMake can generate VS projects for about 20 years
> flawlessly.
> >> > > >>>>> 3. Sometimes even maintainers forget to add test sources to VS
> >> > > projects
> >> > > >>>> [1]
> >> > > >>>>> 4. Currently on TC we build Ignite C++ on windows and linux
> >> > > flawlessly
> >> > > >>>>> using CMake
> >> > > >>>>> 5. VS projects are not backward compatible. We have to add
> >> manually
> >> > > (or
> >> > > >>>>> by
> >> > > >>>>> sed or patch) some dependencies in order to build current VS
> >> > projects
> >> > > >>>>> on
> >> > > >>>>> newer versions of VS.
> >> > > >>>>>
> >> > > >>>>> So I suggest simpy to remove VS projects because of reasons
> I've
> >> > > >>>>> written
> >> > > >>>>> above.
> >> > > >>>>>
> >> > > >>>>> WDYT?
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> [1] --  https://issues.apache.org/jira/browse/IGNITE-15511
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>
> >> > > >>
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Best regards,
> >> > > > Ivan Pavlukhin
> >> > >
> >> > >
> >> >
> >> > --
> >> > Sincerely yours, Ivan Daschinskiy
> >> >
> >>
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: [DISCUSSION] Remove VS project from C++

[Pavel Tupitsyn]

> It makes development much more easier.

How?

> I can hardly imagine how it can be done

1. Get all files in the directory
2. Filter by TestSuite suffix
3. Check if all of them are present in VS files
Am I missing something? We have checks like this for Ignite.NET [1]

[1]
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core.Tests/ProjectFilesTest.cs


On Wed, Sep 15, 2021 at 12:22 PM Ivan Daschinsky 
wrote:

> >> This may become an obstacle for some of the users and I'm not sure how
> it improves anything.
> Please specify more correctly. What is an obstacle? Current VS projects
> (odbc) cannot be build on VC 2015+
> without modification. CMake is an industry standard now.
> >> I'm not sure how it improves anything.
> It makes development much more easier.
>
> >> We can add an automatic check for this (in form of a test).
> I can hardly imagine how it can be done. And even if it is possible, this
> is a sisyphus job.
>
>
> ср, 15 сент. 2021 г. в 12:11, Petr Ivanov :
>
> > +1
> >
> > Let's keep the project clean and on the verge of preferable tech stack.
> >
> >
> > > On 15 Sep 2021, at 12:02, Ivan Pavlukhin  wrote:
> > >
> > > +1 for removing VS project
> > >
> > > 2021-09-15 12:01 GMT+03:00, Nikolay Izhikov :
> > >> +1
> > >>
> > >>> 15 сент. 2021 г., в 11:57, Pavel Tupitsyn 
> > >>> написал(а):
> > >>>
> > >>> -1
> > >>>
> > >>> This may become an obstacle for some of the users and I'm not sure
> how
> > it
> > >>> improves anything.
> > >>>
> > >>>> 3. Sometimes even maintainers forget to add test sources to VS
> > projects
> > >>> [1]
> > >>> We can add an automatic check for this (in form of a test).
> > >>>
> > >>> On Wed, Sep 15, 2021 at 10:28 AM Zhenya Stanilovsky
> > >>>  wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>> completely support !
> > >>>>
> > >>>>> Igniters!
> > >>>>>
> > >>>>> Currently we have CMake build system, that works on Windows, Linux
> > and
> > >>>>> MacOs flawlessly
> > >>>>>
> > >>>>> 1. CMake is supported natively in VS 2019
> > >>>>> 2. CMake can generate VS projects for about 20 years flawlessly.
> > >>>>> 3. Sometimes even maintainers forget to add test sources to VS
> > projects
> > >>>> [1]
> > >>>>> 4. Currently on TC we build Ignite C++ on windows and linux
> > flawlessly
> > >>>>> using CMake
> > >>>>> 5. VS projects are not backward compatible. We have to add manually
> > (or
> > >>>>> by
> > >>>>> sed or patch) some dependencies in order to build current VS
> projects
> > >>>>> on
> > >>>>> newer versions of VS.
> > >>>>>
> > >>>>> So I suggest simpy to remove VS projects because of reasons I've
> > >>>>> written
> > >>>>> above.
> > >>>>>
> > >>>>> WDYT?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> [1] --  https://issues.apache.org/jira/browse/IGNITE-15511
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>
> > >>
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: [DISCUSSION] Remove VS project from C++

[Pavel Tupitsyn]

-1

This may become an obstacle for some of the users and I'm not sure how it
improves anything.

> 3. Sometimes even maintainers forget to add test sources to VS projects
[1]
We can add an automatic check for this (in form of a test).

On Wed, Sep 15, 2021 at 10:28 AM Zhenya Stanilovsky
 wrote:

>
>
> completely support !
>
> >Igniters!
> >
> >Currently we have CMake build system, that works on Windows, Linux and
> >MacOs flawlessly
> >
> >1. CMake is supported natively in VS 2019
> >2. CMake can generate VS projects for about 20 years flawlessly.
> >3. Sometimes even maintainers forget to add test sources to VS projects
> [1]
> >4. Currently on TC we build Ignite C++ on windows and linux flawlessly
> >using CMake
> >5. VS projects are not backward compatible. We have to add manually (or by
> >sed or patch) some dependencies in order to build current VS projects on
> >newer versions of VS.
> >
> >So I suggest simpy to remove VS projects because of reasons I've written
> >above.
> >
> >WDYT?
> >
> >
> >
> >[1] --  https://issues.apache.org/jira/browse/IGNITE-15511
>
>
>
>

Re: [VOTE] Release Apache Ignite 2.11.0 RC2

[Pavel Tupitsyn]

Ilya,

> I'm still confused by this publish/ directory. Why are we shipping
> something which is not for publishing in our binary package? I've also not
> heard of it before.

I think this was introduced in 2.8.0 as part of [1].
It is how the build process works by default and we just copy the entire
bin directory to the package,
while only publish subdir is actually required.

We can probably drop support for legacy (unsupported) .NET versions in the
next release and simplify our build process, I'll look into this.

[1] https://issues.apache.org/jira/browse/IGNITE-11805

On Tue, Sep 14, 2021 at 6:12 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> + 0.5
>
> I have got it to run with the following command:
> % COMPlus_EnableAlternateStackCheck=1 dotnet --fx-version 2.1.30
> Apache.Ignite.dll
>
> I'm still confused by this publish/ directory. Why are we shipping
> something which is not for publishing in our binary package? I've also not
> heard of it before.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 14 сент. 2021 г. в 12:37, Pavel Tupitsyn :
>
> > Ilya,
> >
> > > When I have also installed 3.1 and 2.0
> > Please try .NET SDK 2.1 to run tests.
> > Yes, it went out of support last month, I've created a ticket [1].
> >
> >
> > > dotnet Apache.Ignite.dll
> > That's correct, but should be done
> > in platforms/dotnet/bin/netcoreapp2.0/publish, please try again.
> > We should deal with duplicate binaries, I've created another ticket [2]
> >
> >
> > > Why do we still ship Apache.Ignite.exe if it does not run?
> > It runs on Windows. For Linux we ship Apache.Ignite.dll.
> >
> >
> > I don't think any of this is a blocker for the current release.
> > - Out-of-support SDK only affects those who build from source, and it
> went
> > out of support only recently
> > - Duplicate binaries are there for a long time and no one was complaining
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15504
> > [2] https://issues.apache.org/jira/browse/IGNITE-15505
> >
> > On Tue, Sep 14, 2021 at 11:58 AM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > - 0.5
> > >
> > > Installed DEB package, built binary from source, ran sqlline, built
> .NET.
> > >
> > > I have failed to run .NET tests though.
> > >
> > > If I only install Current .NET, I get the following:
> > > It was not possible to find any compatible framework version
> > > The framework 'Microsoft.NETCore.App', version '2.0.0' was not found.
> > >
> > >
> > >  - The following frameworks were found:
> > >
> > >
> > >  5.0.9 at [/home/ilyak/.dotnet/shared/Microsoft.NETCore.App]
> > >
> > >
> > > You can resolve the problem by installing the specified framework
> and/or
> > > SDK.
> > >
> > >
> > > The specified framework can be found at:
> > >
> > >
> > >  -
> > >
> > >
> >
> https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=2.0.0&arch=x64&rid=ubuntu.21.04-x64
> > >
> > > When I have also installed 3.1 and 2.0, now I see:
> > >
> > > The active test run was aborted. Reason: Test host process crashed : No
> > > usable version of the libssl was found
> > >
> > > It looks like we have to bump the permitted .NET version.
> > >
> > > From the slim binary package I also can't run .NET:
> > >
> > > ~/D/apache-ignite-slim-2.11.0-bin/platforms/dotnet/bin/netcoreapp2.0%
> > > dotnet Apache.Ignite.dll
> > > Error:
> > >  An assembly specified in the application dependencies manifest
> > > (Apache.Ignite.deps.json) was not found:
> > >package: 'System.Configuration.ConfigurationManager', version:
> '4.4.0'
> > >path:
> > 'lib/netstandard2.0/System.Configuration.ConfigurationManager.dll'
> > >
> > > Do we have any instructions on how to run it under Linux? Why do we
> still
> > > ship Apache.Ignite.exe if it does not run?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вт, 14 сент. 2021 г. в 11:28, 18624049226 <18624049...@163.com>:
> > >
> > > > check again, 15503 not cherry-pick, right?
> > > >
> > > > 在 2021/9/14 15:55, S

Re: Deprecating LOCAL cache

[Pavel Tupitsyn]

I agree with Ivan, let's return an error on any attempt to create or use a
LOCAL cache from thin clients.

On Tue, Sep 14, 2021 at 2:25 PM Ivan Daschinsky  wrote:

> I am not about creation per se, but creation from a thin client side.
>
> This feature simply doesn't work as expected, broken and impossible to fix.
> It cannot broke any code, because it was already broken and it is
> impossible to use in production.
> But it still can embarrass newcomers and brings a lot of frustration.
>
> It is much more safe to ban a creation of LOCAL caches from thin clients.
>
> But it can survive for a while for ignite cluster nodes, client and server.
>
> вт, 14 сент. 2021 г. в 14:06, Maxim Muzafarov :
>
> > Ivan,
> >
> > I don't think we should rush with this. Banning the creation of LOCAL
> > caches without a warning through the code sounds not good. Will it be
> > better to do everything in three steps (releases)? 2.12 deprecate,
> > 2.13 forbid new cache creation, 2.14 remove.
> >
> > On Tue, 14 Sept 2021 at 12:09, Ivan Daschinsky 
> > wrote:
> > >
> > > Few thoughts about LOCAL caches on thin client:
> > >
> > > 1. If partition awareness is disabled:
> > > a. Inconsistent behaviour if node to which client is connected goes
> down.
> > > 2. If partition awareness is enabled:
> > > a. For Java and .NET -- same as 1a
> > > b. For C++ and python -- use random routing for caches that are not
> > > PARTITIONED, so inconsistent behaviour from the beginning.
> > >
> > > So I suppose we should ban creation of LOCAL caches from thin client in
> > > 2.12 (fail attempt to create such caches in ClientRequestHandler
> > >
> > > вт, 14 сент. 2021 г. в 11:31, Ivan Daschinsky :
> > >
> > > > >> Unsupported operation exception.
> > > > Binary protocol doesn't have a concept of exception, only error
> status
> > and
> > > > message, but it is just a remark
> > > > I suppose that response with error status and message is ok, but may
> be
> > > > others have different opinion?
> > > >
> > > > >> Removal should happen at 2.13.
> > > > A few thin clients are released separately. I suppose that it is
> > better to
> > > > remove this feature from pyignite a little bit earlier.
> > > >
> > > > вт, 14 сент. 2021 г. в 11:21, Anton Vinogradov :
> > > >
> > > >> > 1. What is expected behaviour if an old thin client requests
> > creation of
> > > >> > LOCAL cache on the newest ignite cluster?
> > > >> Unsupported operation exception.
> > > >>
> > > >> > 2. Should we completely remove LOCAL caches support in thin
> clients
> > > >> (i.e.
> > > >> pyignite) before 2.13 release?
> > > >> Removal should happen at 2.13.
> > > >>
> > > >> On Tue, Sep 14, 2021 at 10:30 AM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > >> 2. 2.13 - complete removal LOCAL caches from codebase.
> > > >> > Let's discuss this step with more details.
> > > >> > 1. What is expected behaviour if an old thin client requests
> > creation of
> > > >> > LOCAL cache on the newest ignite cluster?
> > > >> > 2. Should we completely remove LOCAL caches support in thin
> clients
> > > >> (i.e.
> > > >> > pyignite) before 2.13 release?
> > > >> >
> > > >> > вт, 14 сент. 2021 г. в 10:11, Nikolay Izhikov <
> nizhi...@apache.org
> > >:
> > > >> >
> > > >> > > I proposed the following plan:
> > > >> > >
> > > >> > > 1. 2.12 - deprecation of LOCAL caches.
> > > >> > > 2. 2.13 - complete removal LOCAL caches from codebase.
> > > >> > >
> > > >> > > > 13 сент. 2021 г., в 13:30, Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > > >> > > написал(а):
> > > >> > > >
> > > >> > > > I personally support deprecation, but we should at least have
> a
> > > >> plan.
> > > >> > > > I suppose that putting annotations and removing documentation
> > are
> > > >> not
> > > >> > > > enough.
> > > >> > > >
> > > >> > > >
> > > >> > > > пн, 13 сент. 2021 г. в 13:22, Maxim Muzafarov <
> > mmu...@apache.org>:
> > > >> > > >
> > > >> > > >> Ivan,
> > > >> > > >>
> > > >> > > >> I don't think we can remove LOCAL caches at the nearest time,
> > so
> > > >> there
> > > >> > > >> is no plan for that. I can only imagine a single release that
> > will
> > > >> > > >> contain all the breaking changes we want to apply in 2.x
> > version.
> > > >> > > >>
> > > >> > > >> My point here is only about deprecation:
> > > >> > > >> - there are a lot of motivation points to remove written in
> > this
> > > >> > thread;
> > > >> > > >> - I always hear from the support team that they do not
> > recommend
> > > >> using
> > > >> > > >> local caches;
> > > >> > > >> - I haven't seen any bugs fixed for a long time for local
> > caches
> > > >> > > >> (suppose that we are not maintaining them);
> > > >> > > >>
> > > >> > > >> I just want to make sure that all these points are reflected
> > in the
> > > >> > > >> code base, so propose to mark them as deprecated.
> > > >> > > >>
> > > >> > > >> On Mon, 13 Sept 2021 at 11:29, Ivan Daschinsky <
> > > >> ivanda...@gmail.com>
> > > >> > > >> wrote:
> > 

Re: [VOTE] Release pyignite 0.5.2-rc0

[Pavel Tupitsyn]

> +1
> Отправлено с iPhone

Nikolay, did you test pyignite from your iPhone? Does it work? :)

On Tue, Sep 14, 2021 at 12:22 PM Николай Ижиков  wrote:

> +1
>
> Отправлено с iPhone
>
> > 14 сент. 2021 г., в 11:39, Pavel Tupitsyn 
> написал(а):
> >
> > +1
> >
> > Checked on Ubuntu 20.04, ran a few examples.
> >
> >> On Tue, Sep 14, 2021 at 11:12 AM Ivan Daschinsky 
> >> wrote:
> >>
> >> +1 from me
> >> Checked on windows 10 x86_64 (visual studio 2017) and ubuntu 20.04
> x86_64
> >> and on pythons 3.6, 3.7, 3.8, 3.9 (both win and linux):
> >> 1. Installing from source -- passe
> >> 2. Building wheels from source -- passed
> >> 3. Installing wheels -- passed
> >>
> >> Checked on each steps C module and examples. -- passed
> >>
> >> Checked hashsums and gpg signatures -- passed
> >>
> >> пт, 10 сент. 2021 г. в 19:08, Ivan Daschinsky :
> >>
> >>> Dear Igniters!
> >>>
> >>> Release candidate binaries for subj are uploaded and ready for vote
> >>> You can find them here:
> >>> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.2-rc0
> >>>
> >>> If you follow the link above, you will find source packages (*.tar.gz
> and
> >>> *.zip)
> >>> and binary packages (wheels) for windows (amd64) and linux (x86_64)
> >>> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> >>> Code signing keys can be found here --
> >>> https://downloads.apache.org/ignite/KEYS
> >>> Here you can find instructions how to verify packages
> >>> https://www.apache.org/info/verification.html
> >>>
> >>> You can install binary package for specific version of python using pip
> >>> For example do this on linux for python 3.8
> >>>>> pip install pyignite-0.5.2-cp38-cp38-manylinux1_x86_64.whl
> >>>
> >>> You can build and install package from source using this command:
> >>>>> pip install pyignite-0.5.2.tar.gz
> >>> You can build wheel on your platform using this command:
> >>>>> pip wheel --no-deps pyignite-0.5.2.tar.gz
> >>>
> >>> For building C module, you should have python headers and C compiler
> >>> installed.
> >>> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> >>> In Mac OS X xcode-tools and python from homebrew are the best option.
> >>>
> >>> In order to check whether C module works, use following:
> >>>>> from pyignite import _cutils
> >>>>> print(_cutils.hashcode('test'))
> >>>>> 3556498
> >>>
> >>> You can find documentation here:
> >>>
> >>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/
> >>>
> >>> You can find examples here (to check them, you should start ignite
> >>> locally):
> >>>
> >>>
> >>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/examples.html
> >>> Also, examples can be found in source archive in examples subfolder.
> >>> docker-compose.yml is supplied in order to start ignite quickly. (Use
> >>> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> >>> down` to shut down it)
> >>>
> >>> Release notes:
> >>>
> >>>
> >>
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=a67624a9c141ad5457cdda1a50bd57af5ac62615;hb=1222f29abca4c44a8a2f23e413eafb6acd332e76
> >>>
> >>> Git release tag was created:
> >>>
> >>>
> >>
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.2.rc0
> >>>
> >>> The vote is formal, see voting guidelines
> >>> https://www.apache.org/foundation/voting.html
> >>>
> >>> +1 - to accept pyignite-0.5.2-rc0
> >>> 0 - don't care either way
> >>> -1 - DO NOT accept pyignite-0.5.2-rc0
> >>>
> >>> The vote finishes at 09/15/2021 15:00 UTC
> >>>
> >>
> >>
> >> --
> >> Sincerely yours, Ivan Daschinskiy
> >>
>

Re: [VOTE] Release Apache Ignite 2.11.0 RC2

[Pavel Tupitsyn]

Ilya,

> When I have also installed 3.1 and 2.0
Please try .NET SDK 2.1 to run tests.
Yes, it went out of support last month, I've created a ticket [1].


> dotnet Apache.Ignite.dll
That's correct, but should be done
in platforms/dotnet/bin/netcoreapp2.0/publish, please try again.
We should deal with duplicate binaries, I've created another ticket [2]


> Why do we still ship Apache.Ignite.exe if it does not run?
It runs on Windows. For Linux we ship Apache.Ignite.dll.


I don't think any of this is a blocker for the current release.
- Out-of-support SDK only affects those who build from source, and it went
out of support only recently
- Duplicate binaries are there for a long time and no one was complaining


[1] https://issues.apache.org/jira/browse/IGNITE-15504
[2] https://issues.apache.org/jira/browse/IGNITE-15505

On Tue, Sep 14, 2021 at 11:58 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> - 0.5
>
> Installed DEB package, built binary from source, ran sqlline, built .NET.
>
> I have failed to run .NET tests though.
>
> If I only install Current .NET, I get the following:
> It was not possible to find any compatible framework version
> The framework 'Microsoft.NETCore.App', version '2.0.0' was not found.
>
>
>  - The following frameworks were found:
>
>
>  5.0.9 at [/home/ilyak/.dotnet/shared/Microsoft.NETCore.App]
>
>
> You can resolve the problem by installing the specified framework and/or
> SDK.
>
>
> The specified framework can be found at:
>
>
>  -
>
> https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=2.0.0&arch=x64&rid=ubuntu.21.04-x64
>
> When I have also installed 3.1 and 2.0, now I see:
>
> The active test run was aborted. Reason: Test host process crashed : No
> usable version of the libssl was found
>
> It looks like we have to bump the permitted .NET version.
>
> From the slim binary package I also can't run .NET:
>
> ~/D/apache-ignite-slim-2.11.0-bin/platforms/dotnet/bin/netcoreapp2.0%
> dotnet Apache.Ignite.dll
> Error:
>  An assembly specified in the application dependencies manifest
> (Apache.Ignite.deps.json) was not found:
>package: 'System.Configuration.ConfigurationManager', version: '4.4.0'
>path: 'lib/netstandard2.0/System.Configuration.ConfigurationManager.dll'
>
> Do we have any instructions on how to run it under Linux? Why do we still
> ship Apache.Ignite.exe if it does not run?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 14 сент. 2021 г. в 11:28, 18624049226 <18624049...@163.com>:
>
> > check again, 15503 not cherry-pick, right?
> >
> > 在 2021/9/14 15:55, Shishkov Ilya 写道:
> > >> Should the following two issue be merged into the ignite-2.11 branch?
> > > Hi, as I see, both tickets are fixed in 2.10 and 2.11.
> > >
> > > вт, 14 сент. 2021 г. в 04:13, 18624049226<18624049...@163.com>:
> > >
> > >> Should the following two issue be merged into the ignite-2.11 branch?
> > >>
> > >> https://issues.apache.org/jira/browse/IGNITE-14404
> > >>
> > >> https://issues.apache.org/jira/browse/IGNITE-15503
> > >>
> > >> 在 2021/9/13 17:12, Pavel Tupitsyn 写道:
> > >>> +1
> > >>>
> > >>> Downloaded binary packages, started a cluster with a few nodes,
> tested
> > >> new
> > >>> .NET examples.
> > >>>
> > >>> On Mon, Sep 13, 2021 at 11:21 AM Zhenya Stanilovsky
> > >>>wrote:
> > >>>
> > >>>> Thanks Maxim !
> > >>>> I tries to compare this ver with 2.10 (some performance tests with
> > >>>> persistence and transactional\atomic payload) and seems all ok
> there.
> > >>>> +1 from me.
> > >>>>
> > >>>>> Dear Community,
> > >>>>>
> > >>>>>
> > >>>>> The release candidate for the 2.11 version is ready.
> > >>>>>
> > >>>>>
> > >>>>> I have uploaded a release candidate to:
> > >>>>> https://dist.apache.org/repos/dist/dev/ignite/2.11.0-rc2/
> > >>>>> https://dist.apache.org/repos/dist/dev/ignite/packages_2.11.0-rc2/
> > >>>>>
> > >>>>> The following staging can be used for testing:
> > >>>>>
> > >>
> > https://repository.apache.org/content/repositories/orgapacheignite-1526/
> > >>
> >
> https://www.myget.org/feed/apache-ign

Re: [VOTE] Release pyignite 0.5.2-rc0

[Pavel Tupitsyn]

+1

Checked on Ubuntu 20.04, ran a few examples.

On Tue, Sep 14, 2021 at 11:12 AM Ivan Daschinsky 
wrote:

> +1 from me
> Checked on windows 10 x86_64 (visual studio 2017) and ubuntu 20.04 x86_64
> and on pythons 3.6, 3.7, 3.8, 3.9 (both win and linux):
> 1. Installing from source -- passe
> 2. Building wheels from source -- passed
> 3. Installing wheels -- passed
>
> Checked on each steps C module and examples. -- passed
>
> Checked hashsums and gpg signatures -- passed
>
> пт, 10 сент. 2021 г. в 19:08, Ivan Daschinsky :
>
> > Dear Igniters!
> >
> > Release candidate binaries for subj are uploaded and ready for vote
> > You can find them here:
> > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.2-rc0
> >
> > If you follow the link above, you will find source packages (*.tar.gz and
> > *.zip)
> > and binary packages (wheels) for windows (amd64) and linux (x86_64)
> > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> > Code signing keys can be found here --
> > https://downloads.apache.org/ignite/KEYS
> > Here you can find instructions how to verify packages
> > https://www.apache.org/info/verification.html
> >
> > You can install binary package for specific version of python using pip
> > For example do this on linux for python 3.8
> > >> pip install pyignite-0.5.2-cp38-cp38-manylinux1_x86_64.whl
> >
> > You can build and install package from source using this command:
> > >> pip install pyignite-0.5.2.tar.gz
> > You can build wheel on your platform using this command:
> > >> pip wheel --no-deps pyignite-0.5.2.tar.gz
> >
> > For building C module, you should have python headers and C compiler
> > installed.
> > (i.e. for ubuntu sudo apt install build-essential python3-dev)
> > In Mac OS X xcode-tools and python from homebrew are the best option.
> >
> > In order to check whether C module works, use following:
> > >> from pyignite import _cutils
> > >> print(_cutils.hashcode('test'))
> > >> 3556498
> >
> > You can find documentation here:
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/
> >
> > You can find examples here (to check them, you should start ignite
> > locally):
> >
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/examples.html
> > Also, examples can be found in source archive in examples subfolder.
> > docker-compose.yml is supplied in order to start ignite quickly. (Use
> > `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> > down` to shut down it)
> >
> > Release notes:
> >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=a67624a9c141ad5457cdda1a50bd57af5ac62615;hb=1222f29abca4c44a8a2f23e413eafb6acd332e76
> >
> > Git release tag was created:
> >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.2.rc0
> >
> > The vote is formal, see voting guidelines
> > https://www.apache.org/foundation/voting.html
> >
> > +1 - to accept pyignite-0.5.2-rc0
> > 0 - don't care either way
> > -1 - DO NOT accept pyignite-0.5.2-rc0
> >
> > The vote finishes at 09/15/2021 15:00 UTC
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: Ban Java Streams usage in Ignite 3 code

[Pavel Tupitsyn]

To summarize:
We don't ban streams but use them with caution, avoid on hot paths. The
decision is up to the author and reviewer.

Thanks for the discussion!

On Mon, Sep 13, 2021 at 10:25 PM Pavel Tupitsyn 
wrote:

> Konstantin,
>
> > The performance penalty ... is 25%, not 8 times
> I'm sure you are using a different JDK, I'm on OpenJDK 11 as listed in the
> gist.
>
> Ivan,
>
> > rewrite it using BlackHole
> I did, updated the gist, the results are the same.
>
>
>
> On Thu, Sep 9, 2021 at 11:12 AM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> -1 Let's not ban Java Streams, for the reasons already listed.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 9 сент. 2021 г. в 10:59, Ivan Daschinsky :
>>
>> > Few words about the topic.
>> > There is a disease, quite common in the java community -- it is called
>> the
>> > streamosis.
>> > But, to be honest, afear of streams is also not good.
>> >
>> > As for me, sometimes rewriting code completely with simple loops often
>> > makes it more readable, understandable and usually faster.
>> >
>> > So I am against a complete ban of streams, but I am for using this tool
>> > with caution. Often streams make code ugly and non-readable at all.
>> >
>> >
>> > чт, 9 сент. 2021 г. в 10:50, Ivan Daschinsky :
>> >
>> > > To be honest, Pavel, your benchmark is not quite correct. Please,
>> rewrite
>> > > it using BlackHole
>> > >
>> > > чт, 9 сент. 2021 г. в 10:28, Nikolay Izhikov :
>> > >
>> > >> +1 to ban Streams usage.
>> > >>
>> > >>
>> > >>
>> > >> > 9 сент. 2021 г., в 02:59, Valentin Kulichenko <
>> > >> valentin.kuliche...@gmail.com> написал(а):
>> > >> >
>> > >> > Pavel,
>> > >> >
>> > >> > Quite frankly, I think we used to lean into performance too much.
>> We
>> > >> > generally preferred it over data consistency, project modularity
>> and
>> > >> code
>> > >> > readability. Performance, of course, plays a very important rule in
>> > >> Ignite,
>> > >> > but it's possible to overdo anything.
>> > >> >
>> > >> > There are certainly parts of the project that can benefit from
>> > features
>> > >> > like Stream API, without significant concern over performance. CLI
>> is
>> > an
>> > >> > obvious example, but I'm sure it's not the only one.
>> > >> >
>> > >> > That said, I don't think that banning something is productive. At
>> the
>> > >> same
>> > >> > time, we should make sure we pay more attention to performance
>> during
>> > >> > reviews. Maybe we should have a checklist of major things to look
>> for?
>> > >> Not
>> > >> > as a set of strict rules, but more as a guideline for contributors
>> and
>> > >> > committers.
>> > >> >
>> > >> > We might also want to start benchmarking the code and tracking the
>> > >> progress.
>> > >> >
>> > >> > -Val
>> > >> >
>> > >> > On Wed, Sep 8, 2021 at 12:09 PM Pavel Tupitsyn <
>> ptupit...@apache.org>
>> > >> wrote:
>> > >> >
>> > >> >> Alexander, Ivan,
>> > >> >>
>> > >> >>> not very productive to assume that 100% of your code is
>> > >> >>> on the hot path
>> > >> >>
>> > >> >> That is exactly what we should be doing.
>> > >> >> When I joined Ignite community 6 years ago, this was the prevalent
>> > >> mindset.
>> > >> >>
>> > >> >> I'm not sure which part of Ignite can be considered "not on a hot
>> > >> path".
>> > >> >> Create/alter table (mentioned above) should perform well too.
>> > >> >>
>> > >> >>> measured first and only then optimized
>> > >> >>> https://wiki.c2.com/?OptimizeLater
>> > >> >>
>> > >> >> Extra allocations are a "death by a thousand cuts".
>> > >> >> They add up

Re: Ban Java Streams usage in Ignite 3 code

[Pavel Tupitsyn]

Konstantin,

> The performance penalty ... is 25%, not 8 times
I'm sure you are using a different JDK, I'm on OpenJDK 11 as listed in the
gist.

Ivan,

> rewrite it using BlackHole
I did, updated the gist, the results are the same.



On Thu, Sep 9, 2021 at 11:12 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> -1 Let's not ban Java Streams, for the reasons already listed.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 9 сент. 2021 г. в 10:59, Ivan Daschinsky :
>
> > Few words about the topic.
> > There is a disease, quite common in the java community -- it is called
> the
> > streamosis.
> > But, to be honest, afear of streams is also not good.
> >
> > As for me, sometimes rewriting code completely with simple loops often
> > makes it more readable, understandable and usually faster.
> >
> > So I am against a complete ban of streams, but I am for using this tool
> > with caution. Often streams make code ugly and non-readable at all.
> >
> >
> > чт, 9 сент. 2021 г. в 10:50, Ivan Daschinsky :
> >
> > > To be honest, Pavel, your benchmark is not quite correct. Please,
> rewrite
> > > it using BlackHole
> > >
> > > чт, 9 сент. 2021 г. в 10:28, Nikolay Izhikov :
> > >
> > >> +1 to ban Streams usage.
> > >>
> > >>
> > >>
> > >> > 9 сент. 2021 г., в 02:59, Valentin Kulichenko <
> > >> valentin.kuliche...@gmail.com> написал(а):
> > >> >
> > >> > Pavel,
> > >> >
> > >> > Quite frankly, I think we used to lean into performance too much. We
> > >> > generally preferred it over data consistency, project modularity and
> > >> code
> > >> > readability. Performance, of course, plays a very important rule in
> > >> Ignite,
> > >> > but it's possible to overdo anything.
> > >> >
> > >> > There are certainly parts of the project that can benefit from
> > features
> > >> > like Stream API, without significant concern over performance. CLI
> is
> > an
> > >> > obvious example, but I'm sure it's not the only one.
> > >> >
> > >> > That said, I don't think that banning something is productive. At
> the
> > >> same
> > >> > time, we should make sure we pay more attention to performance
> during
> > >> > reviews. Maybe we should have a checklist of major things to look
> for?
> > >> Not
> > >> > as a set of strict rules, but more as a guideline for contributors
> and
> > >> > committers.
> > >> >
> > >> > We might also want to start benchmarking the code and tracking the
> > >> progress.
> > >> >
> > >> > -Val
> > >> >
> > >> > On Wed, Sep 8, 2021 at 12:09 PM Pavel Tupitsyn <
> ptupit...@apache.org>
> > >> wrote:
> > >> >
> > >> >> Alexander, Ivan,
> > >> >>
> > >> >>> not very productive to assume that 100% of your code is
> > >> >>> on the hot path
> > >> >>
> > >> >> That is exactly what we should be doing.
> > >> >> When I joined Ignite community 6 years ago, this was the prevalent
> > >> mindset.
> > >> >>
> > >> >> I'm not sure which part of Ignite can be considered "not on a hot
> > >> path".
> > >> >> Create/alter table (mentioned above) should perform well too.
> > >> >>
> > >> >>> measured first and only then optimized
> > >> >>> https://wiki.c2.com/?OptimizeLater
> > >> >>
> > >> >> Extra allocations are a "death by a thousand cuts".
> > >> >> They add up here and there, and then there are GC pauses.
> > >> >> This would be hard to "optimize later".
> > >> >>
> > >> >> It is common for perf-oriented projects to avoid some techniques.
> > >> >> For example, LINQ (streams analog from C# with similar perf issues)
> > is
> > >> >> avoided in libraries and compilers [1].
> > >> >>
> > >> >> [1] https://github.com/dotnet/roslyn/blob/main/CONTRIBUTING.md
> > >> >>
> > >> >> On Wed, Sep 8, 2021 at 9:49 PM Valentin Kulichenko <
> > >> >> valentin.kul

Re: Tuple equality in Ignite 3.x

[Pavel Tupitsyn]

Let's clarify column order and nullable columns handling.

Our current API (Table, KvView) does not rely on column order in
user-provided tuples.
Therefore, two tuples with the same set of columns and equal values for
every column may be considered equal.

Tuple a = Tuple.create().set("name", "x").set("id", 1);
table.upsert(a);

Tuple key = Tuple.create().set("id", 1);
Tuple b = table.get(key); // id column goes first

// a was turned into b by Ignite, therefore they are equal.
assertTrue(a.equals(b));


It gets more complicated with nullable columns: should we consider a tuple
without column X and a tuple with column X set to null as equal?





On Fri, Sep 10, 2021 at 3:49 PM Igor Sapego  wrote:

> Sounds very reasonable to me.
>
> +1
>
> Though the default comparator should be implemented very carefully
> as we had issues with comparison of binary objects in 2.x
>
> Best Regards,
> Igor
>
>
> On Thu, Sep 9, 2021 at 4:04 PM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > Tuple in Ignite 3.x is a replacement for BinaryObject in Ignite 2.x.
> > Let's discuss equality and sorting.
> >
> > - We have multiple Tuple implementations, and our API allows custom,
> > user-defined Tuples as well (which can be useful for performance when
> > bridging Ignite with another system or importing the data from
> somewhere).
> > - We don't have equals()/hashCode() overrides, so it is not possible to
> > compare tuples or put them into a Map.
> >
> > Proposal:
> > - Add public TupleComparator implements Comparator, based on the
> > tuple contents (column names and values)
> > - Implement common TupleComparator#hashCode(Tuple t) method that combines
> > hash codes of column names and values
> > - Implement equals(), hashCode(), and Comparable on all built-in tuples,
> > delegate the logic to TupleComparator
> > - Make Tuple extend Compable
> >
> > This way we cover all sorting/comparing/mapping scenarios for built-in
> > tuples and provide reusable code for third-party implementations.
> >
> > Thoughts?
> >
>

Re: [VOTE] Release Apache Ignite 2.11.0 RC2

[Pavel Tupitsyn]

+1

Downloaded binary packages, started a cluster with a few nodes, tested new
.NET examples.

On Mon, Sep 13, 2021 at 11:21 AM Zhenya Stanilovsky
 wrote:

>
>
> Thanks Maxim !
> I tries to compare this ver with 2.10 (some performance tests with
> persistence and transactional\atomic payload) and seems all ok there.
> +1 from me.
>
> >
> >
> >Dear Community,
> >
> >
> >The release candidate for the 2.11 version is ready.
> >
> >
> >I have uploaded a release candidate to:
> >https://dist.apache.org/repos/dist/dev/ignite/2.11.0-rc2/
> >https://dist.apache.org/repos/dist/dev/ignite/packages_2.11.0-rc2/
> >
> >The following staging can be used for testing:
> >https://repository.apache.org/content/repositories/orgapacheignite-1526/
> >
> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
> >
> >Tag name is 2.11.0-rc2:
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.11.0-rc2
> >
> >RELEASE_NOTES:
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.11
> >
> >Complete list of resolved issues:
> >
> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.11%27
> ))
> >
> >DEVNOTES:
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.11
> >
> >
> >Additional checks have been performed (available for users included into
> >the release group on TeamCity).
> >
> >TC [Check RC: Licenses, compile, chksum]
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=6176219&buildTypeId=ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum&tab=buildResultsDiv
> >
> >
> >The vote is formal, see voting guidelines
> >https://www.apache.org/foundation/voting.html
> >
> >+1 - to accept Apache Ignite 2.11.0-rc2
> >0 - don't care either way
> >-1 - DO NOT accept Apache Ignite Ignite 2.11.0-rc2 (explain why)
> >
> >See notes on how to verify release here
> >https://www.apache.org/info/verification.html
> >and
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> >
> >
> >This vote will be open for 4 days until Thu Sep 16, 12:00 2021 (Moscow
> >time).
> >Please, write me down the thread if you need additional time to check
> >the release.
> >
> https://www.timeanddate.com/countdown/vote?iso=20210916T12&p0=166&msg=%5BVOTE%5D+Release+Apache+Ignite+2.10.0+RC2&font=serif&csz=1
>
>
>
>

Re: Replace Map with List and Iterable in KeyValueView Ignite 3 APIs

[Pavel Tupitsyn]

Igor, good point about "pluggable" collections, and Java has Collector
interface for streams.
Instead of replacing Maps with something, we can add overloads for advanced
usage:

putAll(Iterable)
R getAll(Collector)


On Fri, Sep 10, 2021 at 3:07 PM Igor Sapego  wrote:

> I actually agree with Pavel, at least at putAll() part. We require a Map
> from user
> when we do not really need a Map in this method. What we really need here
> is
> an iterable collection of pairs. Can not see why user can not pass for
> example an
> array here.
>
> Now, when we talk about getAll() method it's more complicated, as in many
> cases
> I think a user would expect the returned collection to provide Map methods.
> In C++
> you can deal with it by providing a method that takes a write iterator.
> Using this
> iterator we can put objects in any collection, provided by the user. Maybe
> we can
> achieve something similar using generics in Java, but I believe if we are
> using this
> approach then we still should provide simple method returning Map, because
> to me
> it looks like this is still going to be the most popular scenario.
>
> Best Regards,
> Igor
>
>
> On Fri, Sep 10, 2021 at 9:50 AM Pavel Tupitsyn 
> wrote:
>
> > Val, Alexei - thanks for your replies.
> > Let's keep the Map approach then.
> >
> > Regarding tuple equality - there is another thread [1], please have a
> look.
> >
> >
> >
> https://lists.apache.org/thread.html/r9ed68cd515bffab6df821bbeccb8e44d0e5536000e5e7dd05bd87017%40%3Cdev.ignite.apache.org%3E
> >
> > On Thu, Sep 9, 2021 at 11:21 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > How do we handle the "equality" part in 2.x? The same problem exists
> > there
> > > as well, but we still somehow return a Map.
> > >
> > > Generally, I like Pavel's ideas, but there are a couple of issues with
> > > them. Firstly, Java developers are really used to maps in this context.
> > > Introducing something else might be confusing - it's a significant
> risk.
> > > Secondly, there is no standard Pair class, so we'll have to introduce a
> > > custom one. That said, I would not change this API in Java. In other
> > > languages, however, we can consider this.
> > >
> > > -Val
> > >
> > > On Thu, Sep 9, 2021 at 8:01 AM Alexei Scherbakov <
> > > alexey.scherbak...@gmail.com> wrote:
> > >
> > > > Pavel,
> > > >
> > > > I think the current API looks more natural compared to your proposal.
> > > >
> > > > -1  from my side, see comments below.
> > > >
> > > > чт, 9 сент. 2021 г. в 15:38, Pavel Tupitsyn :
> > > >
> > > > > Igniters,
> > > > >
> > > > > I propose to replace Map with List in getAll and invokeAll,
> and
> > > > > Iterable in putAll APIs of Ignite 3.x KeyValueView.
> > > > >
> > > > > 1. Performance
> > > > > putAll simply iterates over the map, we can easily accept Iterable
> > > > instead.
> > > > > Iterable can be implemented over anything, it can lazily read data
> > > from a
> > > > > file or some other place, instead of allocating a huge collection
> and
> > > > > performing unnecessary hashing.
> > > > >
> > > > > getAll returns a Map, but we don't know if the user code needs a
> map
> > or
> > > > > just wants to iterate over the results, in which case Map is just
> > > > overhead.
> > > > >
> > > >
> > > > "allocating a huge collection" -
> > > > This method is not intended for loading a huge number of entries.
> > > > The allowed map size for the putAll should be limited to some
> > reasonable
> > > > value.
> > > >
> > > > Streaming loading should be addressed in a separate API similar to
> > > > DataStream from Ignite 2.
> > > >
> > > >
> > > > >
> > > > > 2. Equality
> > > > > getAll returns Map, but in many cases, the map will be
> useless
> > > > > because K does not have proper equals()/hashCode() implementation,
> so
> > > > > map.get(key) does not work.
> > > > >
> > > >
> > > > We shouldn't rely on user equals/hashCode while working with
> key-value
> > > API.
> > > > Internally it uses binary representation of a user object for
> > comparison
> > > > purposes.
> > > > The returned map implementation should work in the same way.
> > > >
> > > >
> > > > >
> > > > > Notes:
> > > > > - It is not clear which Pair class to use yet - IgniteBiTuple or
> > > > something
> > > > > else.
> > > > > - Ignite 3 won't deadlock due to putAll entry order, so we don't
> have
> > > to
> > > > > worry about sorting.
> > > > >
> > > > > Thoughts, objections?
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> >
>

Re: Replace Map with List and Iterable in KeyValueView Ignite 3 APIs

[Pavel Tupitsyn]

Val, Alexei - thanks for your replies.
Let's keep the Map approach then.

Regarding tuple equality - there is another thread [1], please have a look.

https://lists.apache.org/thread.html/r9ed68cd515bffab6df821bbeccb8e44d0e5536000e5e7dd05bd87017%40%3Cdev.ignite.apache.org%3E

On Thu, Sep 9, 2021 at 11:21 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> How do we handle the "equality" part in 2.x? The same problem exists there
> as well, but we still somehow return a Map.
>
> Generally, I like Pavel's ideas, but there are a couple of issues with
> them. Firstly, Java developers are really used to maps in this context.
> Introducing something else might be confusing - it's a significant risk.
> Secondly, there is no standard Pair class, so we'll have to introduce a
> custom one. That said, I would not change this API in Java. In other
> languages, however, we can consider this.
>
> -Val
>
> On Thu, Sep 9, 2021 at 8:01 AM Alexei Scherbakov <
> alexey.scherbak...@gmail.com> wrote:
>
> > Pavel,
> >
> > I think the current API looks more natural compared to your proposal.
> >
> > -1  from my side, see comments below.
> >
> > чт, 9 сент. 2021 г. в 15:38, Pavel Tupitsyn :
> >
> > > Igniters,
> > >
> > > I propose to replace Map with List in getAll and invokeAll, and
> > > Iterable in putAll APIs of Ignite 3.x KeyValueView.
> > >
> > > 1. Performance
> > > putAll simply iterates over the map, we can easily accept Iterable
> > instead.
> > > Iterable can be implemented over anything, it can lazily read data
> from a
> > > file or some other place, instead of allocating a huge collection and
> > > performing unnecessary hashing.
> > >
> > > getAll returns a Map, but we don't know if the user code needs a map or
> > > just wants to iterate over the results, in which case Map is just
> > overhead.
> > >
> >
> > "allocating a huge collection" -
> > This method is not intended for loading a huge number of entries.
> > The allowed map size for the putAll should be limited to some reasonable
> > value.
> >
> > Streaming loading should be addressed in a separate API similar to
> > DataStream from Ignite 2.
> >
> >
> > >
> > > 2. Equality
> > > getAll returns Map, but in many cases, the map will be useless
> > > because K does not have proper equals()/hashCode() implementation, so
> > > map.get(key) does not work.
> > >
> >
> > We shouldn't rely on user equals/hashCode while working with key-value
> API.
> > Internally it uses binary representation of a user object for comparison
> > purposes.
> > The returned map implementation should work in the same way.
> >
> >
> > >
> > > Notes:
> > > - It is not clear which Pair class to use yet - IgniteBiTuple or
> > something
> > > else.
> > > - Ignite 3 won't deadlock due to putAll entry order, so we don't have
> to
> > > worry about sorting.
> > >
> > > Thoughts, objections?
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>

Tuple equality in Ignite 3.x

[Pavel Tupitsyn]

Igniters,

Tuple in Ignite 3.x is a replacement for BinaryObject in Ignite 2.x.
Let's discuss equality and sorting.

- We have multiple Tuple implementations, and our API allows custom,
user-defined Tuples as well (which can be useful for performance when
bridging Ignite with another system or importing the data from somewhere).
- We don't have equals()/hashCode() overrides, so it is not possible to
compare tuples or put them into a Map.

Proposal:
- Add public TupleComparator implements Comparator, based on the
tuple contents (column names and values)
- Implement common TupleComparator#hashCode(Tuple t) method that combines
hash codes of column names and values
- Implement equals(), hashCode(), and Comparable on all built-in tuples,
delegate the logic to TupleComparator
- Make Tuple extend Compable

This way we cover all sorting/comparing/mapping scenarios for built-in
tuples and provide reusable code for third-party implementations.

Thoughts?

Replace Map with List and Iterable in KeyValueView Ignite 3 APIs

[Pavel Tupitsyn]

Igniters,

I propose to replace Map with List in getAll and invokeAll, and
Iterable in putAll APIs of Ignite 3.x KeyValueView.

1. Performance
putAll simply iterates over the map, we can easily accept Iterable instead.
Iterable can be implemented over anything, it can lazily read data from a
file or some other place, instead of allocating a huge collection and
performing unnecessary hashing.

getAll returns a Map, but we don't know if the user code needs a map or
just wants to iterate over the results, in which case Map is just overhead.

2. Equality
getAll returns Map, but in many cases, the map will be useless
because K does not have proper equals()/hashCode() implementation, so
map.get(key) does not work.

Notes:
- It is not clear which Pair class to use yet - IgniteBiTuple or something
else.
- Ignite 3 won't deadlock due to putAll entry order, so we don't have to
worry about sorting.

Thoughts, objections?

Re: Sync vs async APIs in Ignite 3

[Pavel Tupitsyn]

Talked to Ivan in private, and came up with the following per-language
summary:

* Java: sync and async
*. NET: only async
* Python: sync and async
* JavaScript: only async (sync is not possible)

Thin clients in other languages are to be discussed.

On Thu, Sep 9, 2021 at 11:49 AM Ivan Daschinsky  wrote:

> >> You can mix them easily.
> This is far from easily (you have already mentioned continuation problem),
> but for i.e. in python it is absolutely not.
> For kotlin it is a little bit easier, but also not fluent and a little bit
> ugly.
>
> чт, 9 сент. 2021 г. в 11:37, Pavel Tupitsyn :
>
> > Ivan,
> >
> > > Pavel, is it really true, that in .NET sync versions of libraries and
> > tools
> > > are completely eliminated?
> >
> > Far from being eliminated, but there is a movement in this direction.
> > For example, HttpClient [1] only provides async variants for most of the
> > methods.
> >
> > Exposing sync wrappers for async methods is not recommended [2].
> > There is a good explanation of potential threading issues there, and it
> can
> > be applied to Java too.
> >
> >
> > > you cannot mix both functions easily
> >
> > You can mix them easily.
> > C#: table.GetAsync(k).Result or
> table.GetAsync(k).GetAwaiter().GetResult()
> > (different exception handling)
> > Java: table.getAsync(k).get()
> >
> > The same thing we do in sync wrapper for async method is now in the user
> > code.
> > Which is better for two reasons:
> > - users won't accidentally use sync API when async API should be used
> > - when they really have to use sync API, potentially dangerous behavior
> is
> > not hidden
> >
> >
> > [1]
> >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
> > [2]
> >
> >
> https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/
> >
> > On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky 
> > wrote:
> >
> > > >> 2. In languages with proper async support (async-await, etc.), we
> can
> > > skip sync API altogether.
> > > It sounds pretty strange, as for me. Usually you cannot mix both
> > functions
> > > easily, it is called blue-red functions problem [1]
> > > In python you definitely cannot do sync over async, for example
> > > (principally can, but nobody do that because of mediocre performance of
> > > that solution). And many developers
> > > still prefer writing code withou asyncio at all.
> > >
> > > Pavel, is it really true, that in .NET sync versions of libraries and
> > tools
> > > are completely eliminated?
> > >
> > > [1] --
> > >
> https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
> > >
> > > ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :
> > >
> > > > To put it another way:
> > > > - true sync operation completes by itself
> > > > - sync-over-async operation requires another thread to complete
> > > >
> > > > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > That's exactly what I have in mind.
> > > > > Yes, we block the user thread, but then we should unblock it by
> > > > completing
> > > > > the future.
> > > > > We can't complete the future from a Netty thread [1], so we'll need
> > > some
> > > > > other thread from some executor.
> > > > > If there are no threads available (because they are blocked by the
> > sync
> > > > > API above), the future won't complete => deadlock.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > > > >
> > > > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com> wrote:
> > > > >
> > > > >> Pavel,
> > > > >>
> > > > >> I might be missing something - could you please elaborate a little
> > > more?
> > > > >>
> > > > >> When I say "sync on top of async", I basically mean that (for
> > example)
> > > > >> &#

Re: Sync vs async APIs in Ignite 3

[Pavel Tupitsyn]

Ivan,

> Pavel, is it really true, that in .NET sync versions of libraries and
tools
> are completely eliminated?

Far from being eliminated, but there is a movement in this direction.
For example, HttpClient [1] only provides async variants for most of the
methods.

Exposing sync wrappers for async methods is not recommended [2].
There is a good explanation of potential threading issues there, and it can
be applied to Java too.


> you cannot mix both functions easily

You can mix them easily.
C#: table.GetAsync(k).Result or table.GetAsync(k).GetAwaiter().GetResult()
(different exception handling)
Java: table.getAsync(k).get()

The same thing we do in sync wrapper for async method is now in the user
code.
Which is better for two reasons:
- users won't accidentally use sync API when async API should be used
- when they really have to use sync API, potentially dangerous behavior is
not hidden


[1]
https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-5.0#methods
[2]
https://devblogs.microsoft.com/pfxteam/should-i-expose-synchronous-wrappers-for-asynchronous-methods/

On Thu, Sep 9, 2021 at 11:16 AM Ivan Daschinsky  wrote:

> >> 2. In languages with proper async support (async-await, etc.), we can
> skip sync API altogether.
> It sounds pretty strange, as for me. Usually you cannot mix both functions
> easily, it is called blue-red functions problem [1]
> In python you definitely cannot do sync over async, for example
> (principally can, but nobody do that because of mediocre performance of
> that solution). And many developers
> still prefer writing code withou asyncio at all.
>
> Pavel, is it really true, that in .NET sync versions of libraries and tools
> are completely eliminated?
>
> [1] --
> https://elizarov.medium.com/how-do-you-color-your-functions-a6bb423d936d
>
> ср, 8 сент. 2021 г. в 22:33, Pavel Tupitsyn :
>
> > To put it another way:
> > - true sync operation completes by itself
> > - sync-over-async operation requires another thread to complete
> >
> > On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn 
> > wrote:
> >
> > > Val,
> > >
> > > That's exactly what I have in mind.
> > > Yes, we block the user thread, but then we should unblock it by
> > completing
> > > the future.
> > > We can't complete the future from a Netty thread [1], so we'll need
> some
> > > other thread from some executor.
> > > If there are no threads available (because they are blocked by the sync
> > > API above), the future won't complete => deadlock.
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
> > >
> > > On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > >> Pavel,
> > >>
> > >> I might be missing something - could you please elaborate a little
> more?
> > >>
> > >> When I say "sync on top of async", I basically mean that (for example)
> > >> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> > >> understanding, it only blocks the user's thread.
> > >>
> > >> Am I wrong? Or you have a different implementation in mind?
> > >>
> > >> -Val
> > >>
> > >> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
> > >> wrote:
> > >>
> > >> > Val,
> > >> >
> > >> > Agree with your points.
> > >> >
> > >> >
> > >> > > async API should be primary
> > >> >
> > >> > It should be noted that all our APIs are inherently async,
> > >> > because thin client is implemented asynchronously.
> > >> >
> > >> >
> > >> > > with the sync version build on top
> > >> >
> > >> > We should document somehow that sync APIs are based on async ones,
> > >> > because this may be dangerous in some use cases.
> > >> >
> > >> > For example, as a user, I may have a thread pool of 4 threads for
> > >> > Ignite-related usage, that is also set as asyncContinuationExecutor
> > [1].
> > >> > Now if I run a lot of concurrent Ignite requests using sync API,
> all 4
> > >> > threads will end up blocked on CompletableFutures.
> > >> > When one of the operations completes, we enqueue the completion to
> > that
> > 

Re: Sync vs async APIs in Ignite 3

[Pavel Tupitsyn]

To put it another way:
- true sync operation completes by itself
- sync-over-async operation requires another thread to complete

On Wed, Sep 8, 2021 at 10:15 PM Pavel Tupitsyn  wrote:

> Val,
>
> That's exactly what I have in mind.
> Yes, we block the user thread, but then we should unblock it by completing
> the future.
> We can't complete the future from a Netty thread [1], so we'll need some
> other thread from some executor.
> If there are no threads available (because they are blocked by the sync
> API above), the future won't complete => deadlock.
>
> [1]
> https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E
>
> On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Pavel,
>>
>> I might be missing something - could you please elaborate a little more?
>>
>> When I say "sync on top of async", I basically mean that (for example)
>> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
>> understanding, it only blocks the user's thread.
>>
>> Am I wrong? Or you have a different implementation in mind?
>>
>> -Val
>>
>> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
>> wrote:
>>
>> > Val,
>> >
>> > Agree with your points.
>> >
>> >
>> > > async API should be primary
>> >
>> > It should be noted that all our APIs are inherently async,
>> > because thin client is implemented asynchronously.
>> >
>> >
>> > > with the sync version build on top
>> >
>> > We should document somehow that sync APIs are based on async ones,
>> > because this may be dangerous in some use cases.
>> >
>> > For example, as a user, I may have a thread pool of 4 threads for
>> > Ignite-related usage, that is also set as asyncContinuationExecutor [1].
>> > Now if I run a lot of concurrent Ignite requests using sync API, all 4
>> > threads will end up blocked on CompletableFutures.
>> > When one of the operations completes, we enqueue the completion to that
>> > same thread pool, but all threads are blocked on sync APIs, resulting
>> in a
>> > deadlock.
>> >
>> > This can be prevented by using a different asyncContinuationExecutor,
>> but
>> > sync API users won't be usually aware of this.
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-15359
>> >
>> > On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
>> > courtney.robin...@hypi.io>
>> > wrote:
>> >
>> > > Hi Val,
>> > >
>> > > I'd highly support an async first API based on CompletionStage
>> > > <
>> > >
>> >
>> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
>> > > >
>> > > or
>> > > its subtypes like CompletableFuture.
>> > > In Ignite 2 we've written a wrapper library around IgniteFuture to
>> > provide
>> > > CompletionStage instead because many of the newer libs we use support
>> > this.
>> > > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper
>> that
>> > we
>> > > wrote to get what you're suggesting here.
>> > >
>> > > Regards,
>> > > Courtney Robinson
>> > > Founder and CEO, Hypi
>> > > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io>
>> > >
>> > > <https://hypi.io>
>> > > https://hypi.io
>> > >
>> > >
>> > > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
>> > > valentin.kuliche...@gmail.com> wrote:
>> > >
>> > > > Igniters,
>> > > >
>> > > > I would like to gather some opinions on whether we want to focus on
>> > sync
>> > > vs
>> > > > async APIs in Ignite 3.
>> > > >
>> > > > Here are some initial considerations that I have:
>> > > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but
>> they
>> > use
>> > > > non-standard IgniteFuture and provide counterintuitive guarantees.
>> In
>> > my
>> > > > experience, they significantly lack usability, and because of that
>> are
>> > > > rarely used.
>> > > > 2. In general, howeve

Re: Sync vs async APIs in Ignite 3

[Pavel Tupitsyn]

Val,

That's exactly what I have in mind.
Yes, we block the user thread, but then we should unblock it by completing
the future.
We can't complete the future from a Netty thread [1], so we'll need some
other thread from some executor.
If there are no threads available (because they are blocked by the sync API
above), the future won't complete => deadlock.

[1]
https://lists.apache.org/thread.html/r528659381d983a177d779f56ef3d7da6fe17eb3504383f5f87727514%40%3Cdev.ignite.apache.org%3E

On Wed, Sep 8, 2021 at 9:40 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> I might be missing something - could you please elaborate a little more?
>
> When I say "sync on top of async", I basically mean that (for example)
> 'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
> understanding, it only blocks the user's thread.
>
> Am I wrong? Or you have a different implementation in mind?
>
> -Val
>
> On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn 
> wrote:
>
> > Val,
> >
> > Agree with your points.
> >
> >
> > > async API should be primary
> >
> > It should be noted that all our APIs are inherently async,
> > because thin client is implemented asynchronously.
> >
> >
> > > with the sync version build on top
> >
> > We should document somehow that sync APIs are based on async ones,
> > because this may be dangerous in some use cases.
> >
> > For example, as a user, I may have a thread pool of 4 threads for
> > Ignite-related usage, that is also set as asyncContinuationExecutor [1].
> > Now if I run a lot of concurrent Ignite requests using sync API, all 4
> > threads will end up blocked on CompletableFutures.
> > When one of the operations completes, we enqueue the completion to that
> > same thread pool, but all threads are blocked on sync APIs, resulting in
> a
> > deadlock.
> >
> > This can be prevented by using a different asyncContinuationExecutor, but
> > sync API users won't be usually aware of this.
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15359
> >
> > On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
> > courtney.robin...@hypi.io>
> > wrote:
> >
> > > Hi Val,
> > >
> > > I'd highly support an async first API based on CompletionStage
> > > <
> > >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> > > >
> > > or
> > > its subtypes like CompletableFuture.
> > > In Ignite 2 we've written a wrapper library around IgniteFuture to
> > provide
> > > CompletionStage instead because many of the newer libs we use support
> > this.
> > > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper
> that
> > we
> > > wrote to get what you're suggesting here.
> > >
> > > Regards,
> > > Courtney Robinson
> > > Founder and CEO, Hypi
> > > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io>
> > >
> > > <https://hypi.io>
> > > https://hypi.io
> > >
> > >
> > > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Igniters,
> > > >
> > > > I would like to gather some opinions on whether we want to focus on
> > sync
> > > vs
> > > > async APIs in Ignite 3.
> > > >
> > > > Here are some initial considerations that I have:
> > > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they
> > use
> > > > non-standard IgniteFuture and provide counterintuitive guarantees. In
> > my
> > > > experience, they significantly lack usability, and because of that
> are
> > > > rarely used.
> > > > 2. In general, however, async execution becomes more and more
> > prominent.
> > > > Something we can't ignore if we want to create a modern framework.
> > > > 3. Still, async support in Java is very limited (especially if
> compared
> > > to
> > > > other languages, like C# for example).
> > > >
> > > > My current position is the following (happy to discuss):
> > > > 1. We should pay more attention to async APIs. As a general rule,
> async
> > > API
> > > > should be primary, with the sync version build on top.
> > > > 2. In languages with proper async support (async-await, etc.), we can
> > > skip
> > > > sync API altogether. As an example of this, you can look at the first
> > > > version of the .NET client [1]. It exposes only async methods, and it
> > > > doesn't look like sync counterparts are really needed.
> > > > 3. In Java (as well as other languages where applicable), we will add
> > > sync
> > > > APIs that simply delegate to async APIs. This will help users to
> avoid
> > > > CompletableFuture if they don't want to use it.
> > > >
> > > > [1] https://github.com/apache/ignite-3/pull/306
> > > >
> > > > Please share your thoughts.
> > > >
> > > > -Val
> > > >
> > >
> >
>

Re: Ban Java Streams usage in Ignite 3 code

[Pavel Tupitsyn]

Alexander, Ivan,

> not very productive to assume that 100% of your code is
> on the hot path

That is exactly what we should be doing.
When I joined Ignite community 6 years ago, this was the prevalent mindset.

I'm not sure which part of Ignite can be considered "not on a hot path".
Create/alter table (mentioned above) should perform well too.

> measured first and only then optimized
> https://wiki.c2.com/?OptimizeLater

Extra allocations are a "death by a thousand cuts".
They add up here and there, and then there are GC pauses.
This would be hard to "optimize later".

It is common for perf-oriented projects to avoid some techniques.
For example, LINQ (streams analog from C# with similar perf issues) is
avoided in libraries and compilers [1].

[1] https://github.com/dotnet/roslyn/blob/main/CONTRIBUTING.md

On Wed, Sep 8, 2021 at 9:49 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> I don't think we should ban anything. Streams API is just a tool in the
> toolbox - it should be used appropriately. It's up to the contributor and
> reviewer(s) to identify whether a particular usage might cause performance
> issues.
>
> -Val
>
> On Wed, Sep 8, 2021 at 8:01 AM Alexander Polovtcev <
> alexpolovt...@gmail.com>
> wrote:
>
> > -1
> > I think that it is not very productive to assume that 100% of your code
> is
> > on the hot path, it would be impossible to write and maintain. Humans are
> > not very good at guessing where the performance bottlenecks are, so the
> > performance of the possible hot paths should be measured first and only
> > then optimized and documented.
> >
> > On Wed, Sep 8, 2021 at 5:53 PM Ivan Pavlukhin 
> wrote:
> >
> > > Does not this trivial strategy work for us?
> > > https://wiki.c2.com/?OptimizeLater
> > >
> > > 2021-09-08 13:52 GMT+03:00, Andrey Gura :
> > > > Agree that any additional object creation on a hot path could be a
> > > > problem. So hot path should not contain stream API and any other
> > > > potentially problem code (e.g. iterator instead of for by index).
> > > >
> > > > On Wed, Sep 8, 2021 at 1:45 PM Pavel Tupitsyn 
> > > wrote:
> > > >>
> > > >> Ok, maybe a total ban is overkill, but right now streams are used
> even
> > > on
> > > >> some hot paths like getAllAsync [1].
> > > >>
> > > >> Another issue is that Collectors.toList() and other variants don't
> > > accept
> > > >> capacity, and we end up with unnecessary reallocations of underlying
> > > >> arrays.
> > > >>
> > > >> [1]
> > > >>
> > >
> >
> https://github.com/apache/ignite-3/blob/1d7d703ff2b18234b15a9a7b03104fbb73388edf/modules/table/src/main/java/org/apache/ignite/internal/table/KVBinaryViewImpl.java#L83
> > > >>
> > > >> On Wed, Sep 8, 2021 at 1:06 PM Konstantin Orlov <
> kor...@gridgain.com>
> > > >> wrote:
> > > >>
> > > >> > Hi!
> > > >> >
> > > >> > Agree with Ivan that it’s an overkill. Code readability and
> > > >> > maintainability should have
> > > >> > the same priority as the performance (with some exceptions of
> > course).
> > > >> >
> > > >> > BTW the result of your benchmark looks quite strange. The
> > performance
> > > >> > penalty on
> > > >> > my laptop (Core i7 9750H, 6 cores up to 4.50 GHz) is 25%, not 8
> > times:
> > > >> >
> > > >> > Benchmark Mode  Cnt  Score Error
> > > >> > Units
> > > >> > JmhIncrementBenchmark.loopSumthrpt   10  32347.819 ± 676.548
> > > >> > ops/ms
> > > >> > JmhIncrementBenchmark.streamSum  thrpt   10  24459.196 ± 610.152
> > > >> > ops/ms
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Regards,
> > > >> > Konstantin Orlov
> > > >> >
> > > >> >
> > > >> > > On 8 Sep 2021, at 12:23, Ivan Bessonov 
> > > wrote:
> > > >> > >
> > > >> > > Hello Igniters,
> > > >> > >
> > > >> > > I object, banning streams is an overkill. I would argue that
> most
> > of
> > > >> > > the
> > >

Re: Ban Java Streams usage in Ignite 3 code

[Pavel Tupitsyn]

Ok, maybe a total ban is overkill, but right now streams are used even on
some hot paths like getAllAsync [1].

Another issue is that Collectors.toList() and other variants don't accept
capacity, and we end up with unnecessary reallocations of underlying arrays.

[1]
https://github.com/apache/ignite-3/blob/1d7d703ff2b18234b15a9a7b03104fbb73388edf/modules/table/src/main/java/org/apache/ignite/internal/table/KVBinaryViewImpl.java#L83

On Wed, Sep 8, 2021 at 1:06 PM Konstantin Orlov  wrote:

> Hi!
>
> Agree with Ivan that it’s an overkill. Code readability and
> maintainability should have
> the same priority as the performance (with some exceptions of course).
>
> BTW the result of your benchmark looks quite strange. The performance
> penalty on
> my laptop (Core i7 9750H, 6 cores up to 4.50 GHz) is 25%, not 8 times:
>
> Benchmark Mode  Cnt  Score Error   Units
> JmhIncrementBenchmark.loopSumthrpt   10  32347.819 ± 676.548  ops/ms
> JmhIncrementBenchmark.streamSum  thrpt   10  24459.196 ± 610.152  ops/ms
>
>
> --
> Regards,
> Konstantin Orlov
>
>
> > On 8 Sep 2021, at 12:23, Ivan Bessonov  wrote:
> >
> > Hello Igniters,
> >
> > I object, banning streams is an overkill. I would argue that most of the
> > code
> > is not on hot paths and that allocations in TLAB don't create much
> pressure
> > on GC.
> >
> > Streams must be used cautiously, developers should know whether they
> > write hot methods or not. And if methods are not hot, code simplicity
> must
> > be
> > the first priority. I don't want Ignite 3 code to look like Ignite 2
> code,
> > where
> > people would iterate over Lists using explicit access by indexes,
> because it
> > saves them a single Iterator allocation. That's absurd.
> >
> > ср, 8 сент. 2021 г. в 11:43, Pavel Tupitsyn :
> >
> >> Igniters,
> >>
> >> Java streams are known to be slower and cause more GC pressure than an
> >> equivalent loop.
> >> Below is a simple filter/map/reduce scenario (code [1]):
> >>
> >> * Benchmark Mode
> Cnt
> >>Score Error   Units
> >>
> >> * StreamVsLoopBenchmark.loopSum thrpt
>   3
> >> 7987.016 ± 293.013  ops/ms
> >> * StreamVsLoopBenchmark.loopSum:·gc.alloc.rate  thrpt
>   3
> >>   ≈ 10⁻⁴MB/sec
> >> * StreamVsLoopBenchmark.loopSum:·gc.count   thrpt
>   3
> >>  ≈ 0counts
> >>
> >> * StreamVsLoopBenchmark.streamSum   thrpt
>   3
> >> 1060.244 ±  36.485  ops/ms
> >> * StreamVsLoopBenchmark.streamSum:·gc.alloc.ratethrpt
>   3
> >>  315.819 ±  10.844  MB/sec
> >> * StreamVsLoopBenchmark.streamSum:·gc.count thrpt
>   3
> >>   55.000counts
> >>
> >> Loop is several times faster and does not allocate at all.
> >>
> >> 1. Performance is one of the most important features of our product.
> >> 2. Most of our APIs will be on the hot path.
> >>
> >> One can argue about performance differences in real-world scenarios,
> >> but increasing GC pressure just to make the code a little bit nicer is
> >> unacceptable.
> >>
> >> I propose to ban streams usage in the codebase (except for the tests).
> >>
> >> Thoughts, objections?
> >>
> >> [1] https://gist.github.com/ptupitsyn/5934bbbf8f92ac4937e534af9386da97
> >>
> >
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
>
>

Ban Java Streams usage in Ignite 3 code

[Pavel Tupitsyn]

Igniters,

Java streams are known to be slower and cause more GC pressure than an
equivalent loop.
Below is a simple filter/map/reduce scenario (code [1]):

 * Benchmark Mode  Cnt
Score Error   Units

 * StreamVsLoopBenchmark.loopSum thrpt3
 7987.016 ± 293.013  ops/ms
 * StreamVsLoopBenchmark.loopSum:·gc.alloc.rate  thrpt3
   ≈ 10⁻⁴MB/sec
 * StreamVsLoopBenchmark.loopSum:·gc.count   thrpt3
  ≈ 0counts

 * StreamVsLoopBenchmark.streamSum   thrpt3
 1060.244 ±  36.485  ops/ms
 * StreamVsLoopBenchmark.streamSum:·gc.alloc.ratethrpt3
  315.819 ±  10.844  MB/sec
 * StreamVsLoopBenchmark.streamSum:·gc.count thrpt3
   55.000counts

Loop is several times faster and does not allocate at all.

1. Performance is one of the most important features of our product.
2. Most of our APIs will be on the hot path.

One can argue about performance differences in real-world scenarios,
but increasing GC pressure just to make the code a little bit nicer is
unacceptable.

I propose to ban streams usage in the codebase (except for the tests).

Thoughts, objections?

[1] https://gist.github.com/ptupitsyn/5934bbbf8f92ac4937e534af9386da97

Re: Sync vs async APIs in Ignite 3

[Pavel Tupitsyn]

Val,

Agree with your points.


> async API should be primary

It should be noted that all our APIs are inherently async,
because thin client is implemented asynchronously.


> with the sync version build on top

We should document somehow that sync APIs are based on async ones,
because this may be dangerous in some use cases.

For example, as a user, I may have a thread pool of 4 threads for
Ignite-related usage, that is also set as asyncContinuationExecutor [1].
Now if I run a lot of concurrent Ignite requests using sync API, all 4
threads will end up blocked on CompletableFutures.
When one of the operations completes, we enqueue the completion to that
same thread pool, but all threads are blocked on sync APIs, resulting in a
deadlock.

This can be prevented by using a different asyncContinuationExecutor, but
sync API users won't be usually aware of this.


[1] https://issues.apache.org/jira/browse/IGNITE-15359

On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson 
wrote:

> Hi Val,
>
> I'd highly support an async first API based on CompletionStage
> <
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> >
> or
> its subtypes like CompletableFuture.
> In Ignite 2 we've written a wrapper library around IgniteFuture to provide
> CompletionStage instead because many of the newer libs we use support this.
> If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper that we
> wrote to get what you're suggesting here.
>
> Regards,
> Courtney Robinson
> Founder and CEO, Hypi
> Tel: ++44 208 123 2413 (GMT+0) 
>
> 
> https://hypi.io
>
>
> On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Igniters,
> >
> > I would like to gather some opinions on whether we want to focus on sync
> vs
> > async APIs in Ignite 3.
> >
> > Here are some initial considerations that I have:
> > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they use
> > non-standard IgniteFuture and provide counterintuitive guarantees. In my
> > experience, they significantly lack usability, and because of that are
> > rarely used.
> > 2. In general, however, async execution becomes more and more prominent.
> > Something we can't ignore if we want to create a modern framework.
> > 3. Still, async support in Java is very limited (especially if compared
> to
> > other languages, like C# for example).
> >
> > My current position is the following (happy to discuss):
> > 1. We should pay more attention to async APIs. As a general rule, async
> API
> > should be primary, with the sync version build on top.
> > 2. In languages with proper async support (async-await, etc.), we can
> skip
> > sync API altogether. As an example of this, you can look at the first
> > version of the .NET client [1]. It exposes only async methods, and it
> > doesn't look like sync counterparts are really needed.
> > 3. In Java (as well as other languages where applicable), we will add
> sync
> > APIs that simply delegate to async APIs. This will help users to avoid
> > CompletableFuture if they don't want to use it.
> >
> > [1] https://github.com/apache/ignite-3/pull/306
> >
> > Please share your thoughts.
> >
> > -Val
> >
>

Re: IEP-78 .NET Thin Client for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

Since there are no more comments, I'm going to merge the first PR later
today and continue working on .NET thin client.
There are separate tickets for CI setup [1] and full API support [2] [3].

[1] https://issues.apache.org/jira/browse/IGNITE-15432
[2] https://issues.apache.org/jira/browse/IGNITE-15430
[3] https://issues.apache.org/jira/browse/IGNITE-15431

On Mon, Sep 6, 2021 at 10:02 PM Pavel Tupitsyn  wrote:

> Val,
>
> Would you like me to start the discussion about sync-over-async in Ignite
> 3 Java APIs, or do you plan to do it yourself?
>
> On Fri, Sep 3, 2021 at 10:10 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Makes sense, thanks!
>>
>> -Val
>>
>> On Fri, Sep 3, 2021 at 2:00 AM Pavel Tupitsyn 
>> wrote:
>>
>> > Hi Val,
>> >
>> > This is a very good point.
>> >
>> > I've looked around blogs, docs, and system APIs, and updated the IEP
>> > accordingly:
>> > For Ignite.NET I propose NOT to add sync methods when the actual
>> > implementation is async:
>> > - It is easy to consume async APIs in C# with async/await keywords
>> (added
>> > in 2012 and widely adopted)
>> > - Most codebases are fully async anyway
>> > - System APIs and popular libraries follow this direction
>> > - Sync-over-async is misleading and can affect performance
>> >
>> >
>> > However, I'm not so sure about Java, where async/await are not present,
>> > overall async usage seems to be rarer, and removing sync methods may
>> become
>> > an obstacle for the users in some cases.
>> > Let's create a separate discussion and see what others think.
>> >
>> > Pavel
>> >
>> > On Thu, Sep 2, 2021 at 10:32 PM Valentin Kulichenko <
>> > valentin.kuliche...@gmail.com> wrote:
>> >
>> > > Hi Pavel,
>> > >
>> > > I've looked at the IEP and the public API - looks good to me.
>> > >
>> > > Quick question - do you plan to add sync methods to the interfaces, or
>> > > you're thinking to only leave async? If the latter, what are the
>> > arguments
>> > > for this? The reason I'm asking is that I'm actually thinking about
>> > > suggesting the same for Java as well (or at least having a discussion
>> > about
>> > > this).
>> > >
>> > > -Val
>> > >
>> > > On Thu, Sep 2, 2021 at 10:08 AM Pavel Tupitsyn 
>> > > wrote:
>> > >
>> > > > Igniters,
>> > > >
>> > > > Please review the IEP [1] and the PoC [2] for .NET Thin Client in
>> > Ignite
>> > > > 3.0.
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
>> > > > [2] https://github.com/apache/ignite-3/pull/306
>> > > >
>> > >
>> >
>>
>

Re: IEP-78 .NET Thin Client for Ignite 3.0

[Pavel Tupitsyn]

Val,

Would you like me to start the discussion about sync-over-async in Ignite 3
Java APIs, or do you plan to do it yourself?

On Fri, Sep 3, 2021 at 10:10 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Makes sense, thanks!
>
> -Val
>
> On Fri, Sep 3, 2021 at 2:00 AM Pavel Tupitsyn 
> wrote:
>
> > Hi Val,
> >
> > This is a very good point.
> >
> > I've looked around blogs, docs, and system APIs, and updated the IEP
> > accordingly:
> > For Ignite.NET I propose NOT to add sync methods when the actual
> > implementation is async:
> > - It is easy to consume async APIs in C# with async/await keywords (added
> > in 2012 and widely adopted)
> > - Most codebases are fully async anyway
> > - System APIs and popular libraries follow this direction
> > - Sync-over-async is misleading and can affect performance
> >
> >
> > However, I'm not so sure about Java, where async/await are not present,
> > overall async usage seems to be rarer, and removing sync methods may
> become
> > an obstacle for the users in some cases.
> > Let's create a separate discussion and see what others think.
> >
> > Pavel
> >
> > On Thu, Sep 2, 2021 at 10:32 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Hi Pavel,
> > >
> > > I've looked at the IEP and the public API - looks good to me.
> > >
> > > Quick question - do you plan to add sync methods to the interfaces, or
> > > you're thinking to only leave async? If the latter, what are the
> > arguments
> > > for this? The reason I'm asking is that I'm actually thinking about
> > > suggesting the same for Java as well (or at least having a discussion
> > about
> > > this).
> > >
> > > -Val
> > >
> > > On Thu, Sep 2, 2021 at 10:08 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > Please review the IEP [1] and the PoC [2] for .NET Thin Client in
> > Ignite
> > > > 3.0.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
> > > > [2] https://github.com/apache/ignite-3/pull/306
> > > >
> > >
> >
>

Re: IEP-78 .NET Thin Client for Ignite 3.0

[Pavel Tupitsyn]

Hi Val,

This is a very good point.

I've looked around blogs, docs, and system APIs, and updated the IEP
accordingly:
For Ignite.NET I propose NOT to add sync methods when the actual
implementation is async:
- It is easy to consume async APIs in C# with async/await keywords (added
in 2012 and widely adopted)
- Most codebases are fully async anyway
- System APIs and popular libraries follow this direction
- Sync-over-async is misleading and can affect performance


However, I'm not so sure about Java, where async/await are not present,
overall async usage seems to be rarer, and removing sync methods may become
an obstacle for the users in some cases.
Let's create a separate discussion and see what others think.

Pavel

On Thu, Sep 2, 2021 at 10:32 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Pavel,
>
> I've looked at the IEP and the public API - looks good to me.
>
> Quick question - do you plan to add sync methods to the interfaces, or
> you're thinking to only leave async? If the latter, what are the arguments
> for this? The reason I'm asking is that I'm actually thinking about
> suggesting the same for Java as well (or at least having a discussion about
> this).
>
> -Val
>
> On Thu, Sep 2, 2021 at 10:08 AM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > Please review the IEP [1] and the PoC [2] for .NET Thin Client in Ignite
> > 3.0.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
> > [2] https://github.com/apache/ignite-3/pull/306
> >
>

IEP-78 .NET Thin Client for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

Please review the IEP [1] and the PoC [2] for .NET Thin Client in Ignite
3.0.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
[2] https://github.com/apache/ignite-3/pull/306

Re: Ignite 3 Java Thin Client configuration: HOCON and ignite-configuration module

[Pavel Tupitsyn]

Val, Ivan - agreed. I'll move on with the existing builder approach.

Thanks!

On Sun, Aug 22, 2021 at 5:03 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> The configuration framework is designed to support dynamic configuration
> changes - I doubt this is needed for the client side. I think we should
> start with simple POJOs or builders.
>
> -Val
>
> On Sat, Aug 21, 2021 at 7:03 AM Ivan Daschinsky 
> wrote:
>
> > As for me, it is very strange purpose and not very practical. I can
> hardly
> > imagine why someone will use it. For example, if I use micronaut or
> spring,
> > this dependency will make me angry
> >
> > сб, 21 авг. 2021 г., 16:13 Pavel Tupitsyn :
> >
> > > Ivan, the purpose is to be able to configure thin client with HOCON.
> > >
> > > On Sat, Aug 21, 2021 at 3:37 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > Hi. As for me, it is quite strange to make thin client dependent on
> > third
> > > > party libraries like hocon parser and so on. What's the purpose of
> > this?
> > > >
> > > > сб, 21 авг. 2021 г., 13:32 Pavel Tupitsyn :
> > > >
> > > > > Igniters,
> > > > >
> > > > > I'd like to discuss Java thin client configuration API in Ignite
> 3.0.
> > > > >
> > > > > On one hand, it would be nice to use codegen approach from
> > > > > ignite-configuration module,
> > > > > and have consistent config APIs across servers and thin clients.
> > > > >
> > > > > On the other hand, that API may seem a bit confusing, because for
> one
> > > > > ClientConfigurationSchema we get ClientConfiguration (mutable),
> > > > ClientView
> > > > > (immutable, name should probably be ClientConfigurationView), and
> > > > > ClientChange for mutations.
> > > > >
> > > > > I've drafted some changes in [1], see [2] for a usage example.
> > > > >
> > > > > Should we follow ignite-configuration approach or create something
> > else
> > > > for
> > > > > the thin client?
> > > > >
> > > > >
> > > > > [1] https://github.com/apache/ignite-3/pull/298
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite-3/blob/a26921666f7bff7c45ae35a2244a2bbb2396b241/modules/client/src/test/java/org/apache/ignite/client/ConfigurationTest.java#L42
> > > > >
> > > >
> > >
> >
>

Re: Ignite 3 Java Thin Client configuration: HOCON and ignite-configuration module

[Pavel Tupitsyn]

Ivan, the purpose is to be able to configure thin client with HOCON.

On Sat, Aug 21, 2021 at 3:37 PM Ivan Daschinsky  wrote:

> Hi. As for me, it is quite strange to make thin client dependent on third
> party libraries like hocon parser and so on. What's the purpose of this?
>
> сб, 21 авг. 2021 г., 13:32 Pavel Tupitsyn :
>
> > Igniters,
> >
> > I'd like to discuss Java thin client configuration API in Ignite 3.0.
> >
> > On one hand, it would be nice to use codegen approach from
> > ignite-configuration module,
> > and have consistent config APIs across servers and thin clients.
> >
> > On the other hand, that API may seem a bit confusing, because for one
> > ClientConfigurationSchema we get ClientConfiguration (mutable),
> ClientView
> > (immutable, name should probably be ClientConfigurationView), and
> > ClientChange for mutations.
> >
> > I've drafted some changes in [1], see [2] for a usage example.
> >
> > Should we follow ignite-configuration approach or create something else
> for
> > the thin client?
> >
> >
> > [1] https://github.com/apache/ignite-3/pull/298
> > [2]
> >
> >
> https://github.com/apache/ignite-3/blob/a26921666f7bff7c45ae35a2244a2bbb2396b241/modules/client/src/test/java/org/apache/ignite/client/ConfigurationTest.java#L42
> >
>

Ignite 3 Java Thin Client configuration: HOCON and ignite-configuration module

[Pavel Tupitsyn]

Igniters,

I'd like to discuss Java thin client configuration API in Ignite 3.0.

On one hand, it would be nice to use codegen approach from
ignite-configuration module,
and have consistent config APIs across servers and thin clients.

On the other hand, that API may seem a bit confusing, because for one
ClientConfigurationSchema we get ClientConfiguration (mutable), ClientView
(immutable, name should probably be ClientConfigurationView), and
ClientChange for mutations.

I've drafted some changes in [1], see [2] for a usage example.

Should we follow ignite-configuration approach or create something else for
the thin client?


[1] https://github.com/apache/ignite-3/pull/298
[2]
https://github.com/apache/ignite-3/blob/a26921666f7bff7c45ae35a2244a2bbb2396b241/modules/client/src/test/java/org/apache/ignite/client/ConfigurationTest.java#L42

Re: [DISCUSSION] Code style for Ignite 3

[Pavel Tupitsyn]

+1, as long as 100% of the rules are checked automatically.

On Fri, Aug 20, 2021 at 4:00 PM Andrey Gura  wrote:

> Looks good to me. But Idea configuration for style check is not
> enough. It helps developers but does not automate style checking.
>
> Checkstyle project provides ready to use config based on Google Code
> Style [1]. I hope it matches well with Idea config and we'll avoid any
> confusing incidents.
>
> Let's start with this convention and adopt it to our needs if needed.
>
> [1]
> https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml
>
> On Fri, Aug 20, 2021 at 12:02 PM Alexei Scherbakov
>  wrote:
> >
> > +1
> >
> > пт, 20 авг. 2021 г. в 10:54, Alexander Polovtcev <
> alexpolovt...@gmail.com>:
> >
> > > Hi, Val. This is an extremely welcome change, thank you!
> > >
> > > On Fri, Aug 20, 2021 at 12:17 AM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Igniters,
> > > >
> > > > I would like to discuss a potential change to the coding guidelines
> for
> > > > Ignite 3. Currently, we're using the existing guidelines inherited
> from
> > > > Ignite 2, which are described here:
> > > > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines
> > > >
> > > > Current guidelines, however, exist for many years and have several
> > > issues.
> > > > They are cumbersome, carry a lot of legacy stuff, and can't be
> automated.
> > > > Every now and then, they seem to cause questions and confusion.
> > > >
> > > > While it's hard to make drastic changes in Ignite 2, we still have a
> > > great
> > > > opportunity to update the guidelines in Ignite 3. I would identify
> two
> > > > major goals here:
> > > >
> > > >1. Simplification. Having too many rules and restrictions tend to
> > > >complicate development rather than providing any value. We should
> come
> > > > up
> > > >with a minimum set of rules and then make amends one by one if
> needed.
> > > >2. The ability for automation. I hold a strong belief that code
> style
> > > >checking has to become a part of the build. Therefore, we need to
> make
> > > > sure
> > > >that any rule that ends up in the guideline can be automatically
> > > > verified.
> > > >
> > > > I propose the following process to define the new guideline:
> > > >
> > > >1. Use Google code style as the starting point:
> > > >https://google.github.io/styleguide/javaguide.html
> > > >2. Replace the 100 column limit with 140. The latter is the value
> we
> > > >already use in Ignite 2, and it seems to be more reasonable, in my
> > > > opinion.
> > > >3. Use 4 spaces block indentation and 8 spaces for continuation
> (as
> > > >opposed to 2 and 4). Nothing wrong with 2 spaces, in my view, but
> 4
> > > > spaces
> > > >should provide a smoother transition, as we're really used to this
> > > > style.
> > > >4. For any other changes, initiate separate discussions going
> forward.
> > > >
> > > > Several reasons why I specifically propose Google style:
> > > >
> > > >1. This is essentially the standard for many projects. I don't
> think
> > > >there is a need for us to reinvent the wheel.
> > > >2. It's minimalistic and developer-friendly. No overcomplication.
> > > >3. (probably the most important) It comes with all the required
> tools
> > > >and configurations for automation (e.g., here is the config for
> IDEA:
> > > >
> > >
> https://github.com/google/error-prone/blob/master/.idea/GoogleStyle.xml
> > > > )
> > > >
> > > > Please let me know what you think. If there are no objections, I will
> > > start
> > > > the process.
> > > >
> > > > -Val
> > > >
> > >
> > >
> > > --
> > > With regards,
> > > Aleksandr Polovtcev
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
>

Re: Ignite 3 async continuation executor

[Pavel Tupitsyn]

Courtney,

Ignite 3 uses CompletableFuture.
Yes, it has thenApplyAsync and other methods which accept an Executor,
but we can't force user code to use those methods and not thenApply.

Thanks for describing your use case with custom thread pools,
we'll figure out how to support that in 3.x



On Fri, Aug 20, 2021 at 9:31 AM Pavel Tupitsyn  wrote:

> Alexander,
>
> > it is not expected that a user may want to specify their own custom
> executor
>
> That would be nice, but I'm not sure if this fits into Ignite 3
> configuration approach.
> I'd like to hear more opinions on this.
>
> On Fri, Aug 20, 2021 at 8:10 AM Courtney Robinson <
> courtney.robin...@hypi.io> wrote:
>
>> Pavel I would really welcome this - when we first started with Ignite we
>> were constantly getting the Ignite threads blocked because we'd perform
>> other work on it.
>>
>> I don't know about the configuration part however because this isn't a
>> static thing I'd argue.
>> Is Ignite 3 still using its own types or is it switching to
>> CompletableFuture
>> <
>> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
>> >
>> ?
>> The key APIs in CompletableFuture (acceptEitherAsync,applyToEitherAsync,
>> handleAsync, thenAcceptASync, thenComposeAsync, whenCompleteAsync) all
>> already accept an Executor argument so returning CompletableFuture solves
>> the problem, it'd just need documentation.
>>
>> If Ignite 3 still uses its own types then I'd suggest what's needed is an
>> argument to accept a custom Executor.
>> We have dedicated pools configured now with custom
>> UncaughtExceptionHandler
>> and ThreadFactory
>> <
>> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadFactory.html
>> >
>> that
>> we have various metrics and customisations around. If the only option is
>> the default ForkJoinPool#commonPool we'd lose this when eventually moving
>> to 3.
>>
>> Regards,
>> Courtney Robinson
>> Founder and CEO, Hypi
>> Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io>
>>
>> <https://hypi.io>
>> https://hypi.io
>>
>>
>> On Thu, Aug 19, 2021 at 5:08 PM Alexander Polovtcev <
>> alexpolovt...@gmail.com>
>> wrote:
>>
>> > Pavel, thanks for the response. Do I understand correctly that it is not
>> > expected that a user may want to specify their own custom executor?
>> >
>> > On Thu, Aug 19, 2021 at 6:55 PM Pavel Tupitsyn 
>> > wrote:
>> >
>> > > Hi Alexander,
>> > >
>> > > To be honest, I'm not sure yet - just getting to know this new
>> > > configuration mechanism and format.
>> > >
>> > > Since we can't use a property of type Executor, we'll have to provide
>> > > predefined values.
>> > > It can either be "bool executeAsyncContinuationsDirectly": false
>> > (default)
>> > > => commonPool, true => Runnable::run,
>> > > or "String asyncContinuationExecutor" which allows two values "direct"
>> > and
>> > > "commonPool".
>> > >
>> > > I'm leaning towards the latter:
>> > >
>> > > {
>> > > "node": {
>> > > "metastorageNodes": [ "node-0" ],
>> > > "asyncContinuationExecutor": "commonPool"
>> > > },
>> > > "network": { ... }
>> > > }
>> > >
>> > >
>> > >
>> > > On Thu, Aug 19, 2021 at 6:29 PM Alexander Polovtcev <
>> > > alexpolovt...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi, Pavel!
>> > > >
>> > > > Can you please provide an example (e.g. HOCON snippet) of how this
>> > > > configuration is going to look like in Ignite 3? Or how is this
>> > property
>> > > > going to be set?
>> > > >
>> > > >
>> > > > On Thu, Aug 19, 2021 at 6:00 PM Pavel Tupitsyn <
>> ptupit...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Igniters,
>> > > > >
>> > > > > I propose to add a configurable async continuation executor for
>> > public
>> > > > APIs
>> > > > > to Ignite 3
>> > > > > like w

Re: Ignite 3 async continuation executor

[Pavel Tupitsyn]

Alexander,

> it is not expected that a user may want to specify their own custom
executor

That would be nice, but I'm not sure if this fits into Ignite 3
configuration approach.
I'd like to hear more opinions on this.

On Fri, Aug 20, 2021 at 8:10 AM Courtney Robinson 
wrote:

> Pavel I would really welcome this - when we first started with Ignite we
> were constantly getting the Ignite threads blocked because we'd perform
> other work on it.
>
> I don't know about the configuration part however because this isn't a
> static thing I'd argue.
> Is Ignite 3 still using its own types or is it switching to
> CompletableFuture
> <
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
> >
> ?
> The key APIs in CompletableFuture (acceptEitherAsync,applyToEitherAsync,
> handleAsync, thenAcceptASync, thenComposeAsync, whenCompleteAsync) all
> already accept an Executor argument so returning CompletableFuture solves
> the problem, it'd just need documentation.
>
> If Ignite 3 still uses its own types then I'd suggest what's needed is an
> argument to accept a custom Executor.
> We have dedicated pools configured now with custom UncaughtExceptionHandler
> and ThreadFactory
> <
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadFactory.html
> >
> that
> we have various metrics and customisations around. If the only option is
> the default ForkJoinPool#commonPool we'd lose this when eventually moving
> to 3.
>
> Regards,
> Courtney Robinson
> Founder and CEO, Hypi
> Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io>
>
> <https://hypi.io>
> https://hypi.io
>
>
> On Thu, Aug 19, 2021 at 5:08 PM Alexander Polovtcev <
> alexpolovt...@gmail.com>
> wrote:
>
> > Pavel, thanks for the response. Do I understand correctly that it is not
> > expected that a user may want to specify their own custom executor?
> >
> > On Thu, Aug 19, 2021 at 6:55 PM Pavel Tupitsyn 
> > wrote:
> >
> > > Hi Alexander,
> > >
> > > To be honest, I'm not sure yet - just getting to know this new
> > > configuration mechanism and format.
> > >
> > > Since we can't use a property of type Executor, we'll have to provide
> > > predefined values.
> > > It can either be "bool executeAsyncContinuationsDirectly": false
> > (default)
> > > => commonPool, true => Runnable::run,
> > > or "String asyncContinuationExecutor" which allows two values "direct"
> > and
> > > "commonPool".
> > >
> > > I'm leaning towards the latter:
> > >
> > > {
> > > "node": {
> > > "metastorageNodes": [ "node-0" ],
> > > "asyncContinuationExecutor": "commonPool"
> > > },
> > > "network": { ... }
> > > }
> > >
> > >
> > >
> > > On Thu, Aug 19, 2021 at 6:29 PM Alexander Polovtcev <
> > > alexpolovt...@gmail.com>
> > > wrote:
> > >
> > > > Hi, Pavel!
> > > >
> > > > Can you please provide an example (e.g. HOCON snippet) of how this
> > > > configuration is going to look like in Ignite 3? Or how is this
> > property
> > > > going to be set?
> > > >
> > > >
> > > > On Thu, Aug 19, 2021 at 6:00 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > I propose to add a configurable async continuation executor for
> > public
> > > > APIs
> > > > > to Ignite 3
> > > > > like we have in Ignite 2.x [1]
> > > > >
> > > > > In short, currently, async APIs return a future to the user code.
> > > > > Continuations like "myCode" in "table.getAsync().thenApply(myCode)"
> > > will
> > > > be
> > > > > executed by the same thread that completes the future, which will
> be
> > a
> > > > > Netty thread or some other Ignite thread.
> > > > >
> > > > > This is dangerous because user code can be blocking or
> long-running,
> > > and
> > > > > system threads become unavailable.
> > > > >
> > > > > Proposal:
> > > > > 1. Add asyncContinuationExecutor configuration property, defaults
> to
> > > > > ForkJoinPool#commonPool - both for server and thin client
> > > > > 2. Use this executor to complete all public API futures
> > > > >
> > > > > This means safe default behavior and a possibility to enable unsafe
> > but
> > > > > faster behavior with Runnable::run executor.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-70%3A+Async+Continuation+Executor
> > > > >
> > > >
> > > >
> > > > --
> > > > With regards,
> > > > Aleksandr Polovtcev
> > > >
> > >
> >
> >
> > --
> > With regards,
> > Aleksandr Polovtcev
> >
>

Re: Ignite 3 async continuation executor

[Pavel Tupitsyn]

Hi Alexander,

To be honest, I'm not sure yet - just getting to know this new
configuration mechanism and format.

Since we can't use a property of type Executor, we'll have to provide
predefined values.
It can either be "bool executeAsyncContinuationsDirectly": false (default)
=> commonPool, true => Runnable::run,
or "String asyncContinuationExecutor" which allows two values "direct" and
"commonPool".

I'm leaning towards the latter:

{
"node": {
"metastorageNodes": [ "node-0" ],
"asyncContinuationExecutor": "commonPool"
},
"network": { ... }
}



On Thu, Aug 19, 2021 at 6:29 PM Alexander Polovtcev 
wrote:

> Hi, Pavel!
>
> Can you please provide an example (e.g. HOCON snippet) of how this
> configuration is going to look like in Ignite 3? Or how is this property
> going to be set?
>
>
> On Thu, Aug 19, 2021 at 6:00 PM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > I propose to add a configurable async continuation executor for public
> APIs
> > to Ignite 3
> > like we have in Ignite 2.x [1]
> >
> > In short, currently, async APIs return a future to the user code.
> > Continuations like "myCode" in "table.getAsync().thenApply(myCode)" will
> be
> > executed by the same thread that completes the future, which will be a
> > Netty thread or some other Ignite thread.
> >
> > This is dangerous because user code can be blocking or long-running, and
> > system threads become unavailable.
> >
> > Proposal:
> > 1. Add asyncContinuationExecutor configuration property, defaults to
> > ForkJoinPool#commonPool - both for server and thin client
> > 2. Use this executor to complete all public API futures
> >
> > This means safe default behavior and a possibility to enable unsafe but
> > faster behavior with Runnable::run executor.
> >
> > Thoughts?
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-70%3A+Async+Continuation+Executor
> >
>
>
> --
> With regards,
> Aleksandr Polovtcev
>

Ignite 3 async continuation executor

[Pavel Tupitsyn]

Igniters,

I propose to add a configurable async continuation executor for public APIs
to Ignite 3
like we have in Ignite 2.x [1]

In short, currently, async APIs return a future to the user code.
Continuations like "myCode" in "table.getAsync().thenApply(myCode)" will be
executed by the same thread that completes the future, which will be a
Netty thread or some other Ignite thread.

This is dangerous because user code can be blocking or long-running, and
system threads become unavailable.

Proposal:
1. Add asyncContinuationExecutor configuration property, defaults to
ForkJoinPool#commonPool - both for server and thin client
2. Use this executor to complete all public API futures

This means safe default behavior and a possibility to enable unsafe but
faster behavior with Runnable::run executor.

Thoughts?


[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-70%3A+Async+Continuation+Executor

Re: Storing Teamcity projects settings in Version Control

[Pavel Tupitsyn]

Ivan,

1. It will be clean. It will actually be better: good to know when build
failure is caused by a config change, right?

2. Can you please provide an example in a well-known open-source project
other than Ignite?

Btw, I was always against moving thin clients into separate repos. Monorepo
approach in some large companies is used for a reason.
Dealing with multiple repositories is always hard.


On Tue, Aug 17, 2021 at 6:23 PM Ivan Daschinsky  wrote:

> 1. Clean commit history (As one developer said, git history is an api)
> 2. We have separate thin client repos -- but TC thin client build depends
> on ignite build also.
>
>
> вт, 17 авг. 2021 г. в 18:08, Pavel Tupitsyn :
>
> > Ivan,
> >
> > > I'm sorry, but what about storing TC configs in separate repo?
> > What are the pros of this approach? What do we gain?
> > Separate repo always adds friction, and it is not clear how to handle
> > config changes that are tied to code changes.
> >
> > > It is quite common approach.
> > Can you provide an example of an open-source project with this approach?
> >
> > On Tue, Aug 17, 2021 at 6:05 PM Ivan Daschinsky 
> > wrote:
> >
> > > I'm sorry, but what about storing TC configs in separate repo?
> > > It is quite common approach.
> > >
> > > вт, 17 авг. 2021 г. в 17:33, Pavel Tupitsyn :
> > >
> > > > Anton,
> > > >
> > > > > This will kill repo history.
> > > > > You'll see dozens of TC config updates vs a single Ignite fix
> > > >
> > > > Not really.
> > > > I'm not suggesting something crazy, this is the modern way to do
> CI/CD
> > > > - see GitHub actions, Azure pipelines, etc - you write a config and
> > store
> > > > it in Git.
> > > >
> > > > > Where are you going to apply configs, do you have your own TC? ;)
> > > >
> > > > Maybe I do. That's the point, no matter how many TCs we have, all of
> > them
> > > > will use the same configs from the repo.
> > > >
> > > > On Tue, Aug 17, 2021 at 5:07 PM Petr Ivanov 
> > wrote:
> > > >
> > > > > After initial setup, there won't be lot's of changes, at least for
> > PRs
> > > > > there will be single commit with both fix and TC changes.
> > > > >
> > > > >
> > > > > > On 17 Aug 2021, at 13:05, Anton Vinogradov 
> wrote:
> > > > > >
> > > > > > This will kill repo history.
> > > > > > You'll see dozens of TC config updates vs a single Ignite fix
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: Storing Teamcity projects settings in Version Control

[Pavel Tupitsyn]

Ivan,

> I'm sorry, but what about storing TC configs in separate repo?
What are the pros of this approach? What do we gain?
Separate repo always adds friction, and it is not clear how to handle
config changes that are tied to code changes.

> It is quite common approach.
Can you provide an example of an open-source project with this approach?

On Tue, Aug 17, 2021 at 6:05 PM Ivan Daschinsky  wrote:

> I'm sorry, but what about storing TC configs in separate repo?
> It is quite common approach.
>
> вт, 17 авг. 2021 г. в 17:33, Pavel Tupitsyn :
>
> > Anton,
> >
> > > This will kill repo history.
> > > You'll see dozens of TC config updates vs a single Ignite fix
> >
> > Not really.
> > I'm not suggesting something crazy, this is the modern way to do CI/CD
> > - see GitHub actions, Azure pipelines, etc - you write a config and store
> > it in Git.
> >
> > > Where are you going to apply configs, do you have your own TC? ;)
> >
> > Maybe I do. That's the point, no matter how many TCs we have, all of them
> > will use the same configs from the repo.
> >
> > On Tue, Aug 17, 2021 at 5:07 PM Petr Ivanov  wrote:
> >
> > > After initial setup, there won't be lot's of changes, at least for PRs
> > > there will be single commit with both fix and TC changes.
> > >
> > >
> > > > On 17 Aug 2021, at 13:05, Anton Vinogradov  wrote:
> > > >
> > > > This will kill repo history.
> > > > You'll see dozens of TC config updates vs a single Ignite fix
> > >
> > >
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: Storing Teamcity projects settings in Version Control

[Pavel Tupitsyn]

Anton,

> This will kill repo history.
> You'll see dozens of TC config updates vs a single Ignite fix

Not really.
I'm not suggesting something crazy, this is the modern way to do CI/CD
- see GitHub actions, Azure pipelines, etc - you write a config and store
it in Git.

> Where are you going to apply configs, do you have your own TC? ;)

Maybe I do. That's the point, no matter how many TCs we have, all of them
will use the same configs from the repo.

On Tue, Aug 17, 2021 at 5:07 PM Petr Ivanov  wrote:

> After initial setup, there won't be lot's of changes, at least for PRs
> there will be single commit with both fix and TC changes.
>
>
> > On 17 Aug 2021, at 13:05, Anton Vinogradov  wrote:
> >
> > This will kill repo history.
> > You'll see dozens of TC config updates vs a single Ignite fix
>
>

Re: Storing Teamcity projects settings in Version Control

[Pavel Tupitsyn]

Dmitry, Petr,

I think TC config should be stored in the same repo as the corresponding
code (2.x config in 2.x repo, 3.x in 3.x, etc).

Changes and updates to build scripts and project structure often come
together with changes to TC configuration,
it would be great to be able to test them by simply creating a new branch
in a single repo.

On Mon, Aug 16, 2021 at 10:17 PM Petr Ivanov  wrote:

> Hi, Dmitry.
>
>
> I think we should start with adding current projects as-is in form of
> autogenerated Kotlin code, but continue to use UI for editing.
> Later at some point we should replace autogenerated code with valid one
> (this can be done configuration by configuration), that will allow use the
> power of DSL and programming language to create dynamic configuration and
> use other benefits.
>
> Also, while I was thinking about it, I wondered — should we add whole TC
> config to single repository, or divide per project (AI 2.x, AI 3.x,
> Extensions, Thin Clients, etc.) and add corresponding TeamCity
> configuration as part of the project into the same repository.
> TeamCity configuration is added in form of maven project in .teamcity
> directory in the project's root and will not interfere with the project
> itself. Also this scheme is preferable due to linear development cycle (no
> parallel versions in development for each project).
>
> I think I am ready to drive this initiative when and if we agree on this
> matter and will discuss all the details of such migration.
>
>
> > On 16 Aug 2021, at 19:36, Dmitry Pavlov  wrote:
> >
> > Hi Igniters,
> >
> > Once there was a discussion about placing our build configurations (TC)
> settings in a VSC repository. This idea was suggested because we wanted to
> validate internals of configs using IDE/text editor.
> >
> >
> https://www.jetbrains.com/help/teamcity/storing-project-settings-in-version-control.html
> >
> > Now, with introduction of a second instance this idea migth be useful
> twice, we'll be able to
> > - view, track history, track authorship (? I'm not 100% sure, it is to
> be checked)
> > - use VSC as a signle source of thurth
> > - we can remove TC Bot's code for validating all configs using REST
> >
> > How does that sound to you? Is it better to look at Kotlin DSL? Or could
> XML be enought?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
>
>

Re: IGNITE-15256 request for review

[Pavel Tupitsyn]

Hi Ivan,

Thank you for your contribution!
Please see my comments in JIRA.

On Wed, Aug 11, 2021 at 5:54 PM Mirza Aliev  wrote:

> Helo Ivan! Thank you for your effort!
>
> Here [1] you can find info about the process of contributing to Apache
> Ignite.
> As Krill said before, you need to get "green visa", which means that you
> need to run TeamCity bot and receive no blockers (see "Submitting for
> Review" section here [1])
>
> Feel free to ask questions, if something is unclear.
>
> [1] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>
> ср, 11 авг. 2021 г. в 16:52, ткаленко кирилл :
>
> > Hi, in the beginning it is worth getting a green visa.
> >
> > 11.08.2021, 15:21, "Ivan Fedorenkov" :
> > > Dear Ignite Devs,
> > >
> > > I would like to contribute a fix that addresses a bug associated with
> > > registration of user types by Ignite Java Thin Client. Could someone
> > please
> > > review the code? https://github.com/apache/ignite/pull/9307
> > >
> > > Problem description: when Java Thin Client reestablishes connection to
> > > Ignite and invokes a Service API method with Externalizable input
> > parameter
> > > server node may throw ClassNotFoundException and client marks the
> > > connection as broken and tries to open a new one.
> > >
> > > Root cause: Ignite server nodes may lose all the registered user types
> > > after restart if Ignite Working Dir gets deleted or cluster moves to
> > > another set of hosts. Clients, at the same time, believe that they have
> > > already registered their user types and they skip registration after
> > > reconnecting.
> > >
> > > Please note that this is my first PR into Ignite repo and I may easily
> > miss
> > > some important details.
> > >
> > > Best regards,
> > > Ivan Fedorenkov
> >
>

[ANNOUNCE] Welcome Alexander Shapkin as a new committer

[Pavel Tupitsyn]

The Project Management Committee (PMC) for Apache Ignite has invited
Alexander Shapkin to become a committer and we are pleased to announce that
he has accepted.

Alexander is one of the most active user supporters and has contributed
new features and improvements to the code.

Being a committer enables easier contribution to the project since there is
no need to go via the patch submission process. This should enable better
productivity.

Please join me in welcoming Alexander, and congratulating him on the new role
in
the Apache Ignite Community!

Best Regards,
Pavel Tupitsyn

Re: [DISCUSSION] Send documentation feedback notifications to dev list

[Pavel Tupitsyn]

I agree, let's keep the dev list clean.
Automated notifications of any kind should not be sent to dev@i.a.o.

PS Ivan, links 2 and 3 are the same.

On Mon, Aug 9, 2021 at 8:54 AM Ivan Pavlukhin  wrote:

> Igniters,
>
> Recently documentation feedback notifications were set up. And
> currently destination for such notifications is dev list
> dev@ignite.apache.org. It was announced in [1]. A notification message
> itself looks as [2].
>
> Let's discuss if we all are fine with this notifications on dev list
> as much effort was spent for keeping dev list clean, e.g. [3]. For me
> personally these notifications are some kind of "noice" as I read the
> list on spare time.
>
> Please share your opinion! Formal vote will be started if needed.
>
> [1]
> https://lists.apache.org/thread.html/rf6b0ee140239ae119c97fe4022316b4e2f5c9f06a677bace5033e313%40%3Cdev.ignite.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/rbd7c06f76559f46bc2872484529bc4fa1c503d705b6cc379156f0bb0%40%3Cdev.ignite.apache.org%3E
> [3]
> https://lists.apache.org/thread.html/rbd7c06f76559f46bc2872484529bc4fa1c503d705b6cc379156f0bb0%40%3Cdev.ignite.apache.org%3E
>
> --
>
> Best regards,
> Ivan Pavlukhin
>

Re: Secondary TeamCity instance: ci2.ignite.apache.org

[Pavel Tupitsyn]

Dmitry,

This is awesome!

> tc.i.a.o or ci2.i.a.o
My vote for ci2.i.a.o

On Tue, Aug 3, 2021 at 6:33 PM Dmitry Pavlov  wrote:

> Hi Igniters,
>
> I'm glad to announce that SberTech made an internal aggreement to sponsor
> a computing power to provide backup TeamCity instance. This instance is
> intended to be a geo-independent, secondary instance with availablility for
> community members.
>
> Thanks to JetBrains for providing license keys for TeamCity as their part
> of opensource sponsoring program.
>
> Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o. Please
> share your vision what would be most obvious.
>
> After a private discussions with Vitaliy Osipov and Petr Ivanov, I do
> believe that it would be possible to preserve current registered users.
> Build configurations are to be the same for the secondary instance.
> Technical details on how is could be achieved will be available later (most
> likely we'll start a sepearate thread to dicuss that).
>
> You are more than welcome to be an early adopters.
>
> Sincerely,
> Dmitriy Pavlov
>
>

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Pavel Tupitsyn]

Igniters,

I've made the changes to the Tuple interface, please have a look:

https://github.com/apache/ignite-3/pull/245
https://issues.apache.org/jira/browse/IGNITE-14342

On Thu, Jul 1, 2021 at 12:25 PM Pavel Tupitsyn  wrote:

> Val, agreed.
> Let's add length(), value(index), and Iterable to the Tuple interface.
>
> I've updated the ticket:
> https://issues.apache.org/jira/browse/IGNITE-14342
>
> On Wed, Jun 30, 2021 at 11:17 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Pavel,
>>
>> Thanks for your response, makes sense.
>>
>> Regarding the values() method: I would instead add the required methods to
>> the Tuple itself. E.g., it can implement Iterable, and additionally have
>> the value(int index) method, plus anything else that we might need.
>> I don't like returning a collection, because in Java it's a much wider
>> API.
>> We will end up returning an implementation that throws
>> UnsupportedOperationException for most of the methods. I know this
>> approach
>> is not uncommon in the Java world, but this doesn't make it good.
>> In .NET and other languages, we can use other abstractions, of course.
>> Every platform has its own specifics and practices, so APIs don't have to
>> be identical.
>>
>> -Val
>>
>> On Wed, Jun 30, 2021 at 7:44 AM Pavel Tupitsyn 
>> wrote:
>>
>> > Hi Andrey,
>> >
>> > > This will force us to bother about interfaces/contracts and
>> compatibility
>> > > aspects in the future
>> >
>> > Schemas and versions are part of thin client wire protocol.
>> > This protocol is a public API - we'll have to care about compatibility
>> > anyway.
>> >
>> > Schema evolution is an important feature that users should understand.
>> > I see no reason to hide it from the API.
>> >
>> >
>> > > We already have a ticket [1]...
>> > > Will 'idx -> column' mapping only be enough for your purposes?
>> >
>> > The ticket sounds reasonable, but there are no API details.
>> > At the very least we need public access to column names, types, and
>> > indexes.
>> > I propose something like this:
>> >
>> > interface Tuple {
>> > TupleSchema getSchema();
>> > }
>> >
>> > class TupleSchema {
>> > int getVersion();
>> > List getColumns();
>> > }
>> >
>> > class TupleSchemaColumn {
>> > int index;
>> > String name;
>> > String typeName;
>> > bool isKey;
>> > }
>> >
>> > On Wed, Jun 30, 2021 at 11:05 AM Andrey Mashenkov <
>> > andrey.mashen...@gmail.com> wrote:
>> >
>> > > Hi Pavel,
>> > >
>> > > 2. Schema and its version are internal things and shouldn't be
>> exposed,
>> > > otherwise, eventually, it will lead to the user will manage schemas
>> > > manually on his side for some purposes.
>> > > This will force us to bother about interfaces/contracts and
>> compatibility
>> > > aspects in the future with uncertain benefits for a user.
>> > >
>> > > As SchemaDescriptor is an internal API and the user expects a public
>> API.
>> > > I don't think we want to convert the descriptor back into public API
>> > terms
>> > > and it may be impossible for some data.
>> > >
>> > > We already have a ticket [1] to support accessing a column by an index
>> > and
>> > > exposing some metadata.
>> > > Will 'idx -> column' mapping only be enough for your purposes?
>> > >
>> > > > int Tuple.columnIndex(String)
>> > >
>> > >
>> > >
>> > >  [1] https://issues.apache.org/jira/browse/IGNITE-14342
>> > >
>> > > On Wed, Jun 30, 2021 at 9:34 AM Pavel Tupitsyn 
>> > > wrote:
>> > >
>> > > > Hi Val,
>> > > >
>> > > > Thanks for the comments, please see below:
>> > > >
>> > > >
>> > > > > Users will identify tables using names, they will never use IDs
>> > > >
>> > > > Ok, let's keep it this way.
>> > > >
>> > > >
>> > > > > Sounds like the Tuple should implement Iterable.
>> > > >
>> > > > I don't think Iterable is enough.
>> >

Re: [VOTE] Release pyignite 0.5.1-rc0

[Pavel Tupitsyn]

+1

On Mon, Jul 26, 2021 at 11:36 AM Igor Sapego  wrote:

> +1 from me
>
> Best Regards,
> Igor
>
>
> On Fri, Jul 23, 2021 at 3:32 PM Ivan Daschinsky 
> wrote:
>
> > +1 From me
> > 1. Checked binary packages, c module and examples on windows 10 amd64 for
> > pythons 3.6, 3.7, 3.8, 3.9
> > 2. Checked binary packages, c module and examples on ubuntu 20.04 amd64
> for
> > pythons 3.6, 3.7, 3.8, 3.9
> > 3. Checked source installation and building binary packages on ubuntu
> 20.04
> > amd 64 for pythons 3.6, 3.7, 3.8, 3.9
> > 4. Checked documentation on
> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
> > 5. Checked sha512 checksums and gpg signatures (signed by Igor Sapego
> (CODE
> > SIGNING KEY)  5C10 A072 2D94 7727 923C  98B5 AF35
> DBD9
> > 58FE 8DC5)
> > key is inside https://downloads.apache.org/ignite/KEYS)
> >
> > пт, 23 июл. 2021 г. в 13:52, Ivan Daschinsky :
> >
> > > The voting finishes at 07/27/2021 12:00 UTC
> > >
> > > пт, 23 июл. 2021 г. в 13:49, Ivan Daschinsky :
> > >
> > >> Dear Igniters!
> > >>
> > >> Release candidate binaries for subj are uploaded and ready for vote
> > >> You can find them here:
> > >> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.1-rc0
> > >>
> > >> If you follow the link above, you will find source packages (*.tar.gz
> > and
> > >> *.zip)
> > >> and binary packages (wheels) for windows (amd64) and linux (x86_64)
> > >> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> > >> Code signing keys can be found here --
> > >> https://downloads.apache.org/ignite/KEYS
> > >> Here you can find instructions how to verify packages
> > >> https://www.apache.org/info/verification.html
> > >>
> > >> You can install binary package for specific version of python using
> pip
> > >> For example do this on linux for python 3.8
> > >> >> pip install pyignite-0.5.1-cp38-cp38-manylinux1_x86_64.whl
> > >>
> > >> You can build and install package from source using this command:
> > >> >> pip install pyignite-0.5.1.tar.gz
> > >> You can build wheel on your platform using this command:
> > >> >> pip wheel --no-deps pyignite-0.5.1.tar.gz
> > >>
> > >> For building C module, you should have python headers and C compiler
> > >> installed.
> > >> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> > >> In Mac OS X xcode-tools and python from homebrew are the best option.
> > >>
> > >> In order to check whether C module works, use following:
> > >> >> from pyignite import _cutils
> > >> >> print(_cutils.hashcode('test'))
> > >> >> 3556498
> > >>
> > >> You can find documentation here:
> > >>
> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
> > >>
> > >> You can find examples here (to check them, you should start ignite
> > >> locally):
> > >>
> > >>
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0/examples.html
> > >> Also, examples can be found in source archive in examples subfolder.
> > >> docker-compose.yml is supplied in order to start ignite quickly. (Use
> > >> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> > >> down` to shut down it)
> > >>
> > >> Release notes:
> > >>
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=c6cbd419684cd4a97485707471bac84957b42891;hb=b48dd5dec37064b458031358c394789d15a756fc
> > >>
> > >> Git release tag was created:
> > >>
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.1.rc0
> > >>
> > >> The vote is formal, see voting guidelines
> > >> https://www.apache.org/foundation/voting.html
> > >>
> > >> +1 - to accept pyignite-0.5.1-rc0
> > >> 0 - don't care either way
> > >> -1 - DO NOT accept pyignite-0.5.1-rc0
> > >>
> > >>
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

The initial implementation is ready for review.
To limit the PR size, I've only implemented insert and get operations.

https://issues.apache.org/jira/browse/IGNITE-14970
https://github.com/apache/ignite-3/pull/191

On Wed, Jul 7, 2021 at 1:56 PM Pavel Tupitsyn  wrote:

> Thanks everyone, I've moved the proposal to Active status.
> Working on the implementation.
>
> On Tue, Jul 6, 2021 at 10:31 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> The proposal looks good to me.
>>
>> -Val
>>
>> On Tue, Jul 6, 2021 at 2:24 AM Ivan Daschinsky 
>> wrote:
>>
>> > I suppose, that general idea is great. Some details are missing, but I
>> > suppose during implementation of clients we will add more details and
>> > redefine some parts.
>> >
>> > вт, 6 июл. 2021 г., 09:59 Pavel Tupitsyn :
>> >
>> > > Ivan, Val, and others - are there any open objections or questions?
>> > > Can we accept the proposal?
>> > >
>> > > On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn 
>> > > wrote:
>> > >
>> > > > Igniters,
>> > > >
>> > > > I've updated the IEP to support "Live Schema" [1] from IEP-54.
>> > > > Some operations now have schemaless variants, where tuples are
>> > serialized
>> > > > as maps (String -> val).
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
>> > > >
>> > > > On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky > >
>> > > > wrote:
>> > > >
>> > > >> Val, my understanding about it was exactly the same as yours, but,
>> > > again,
>> > > >> I
>> > > >> heard a different opinion.
>> > > >>
>> > > >> But nevertheless, binary protocol should not be about objects,
>> records
>> > > aka
>> > > >> tuples are the best varii, simple and powerful.
>> > > >>
>> > > >> As for me, I want to take part in implementing python and golang
>> thin
>> > > >> clients for ignite 3, so consider my remarks using this info. I am
>> not
>> > > >> just
>> > > >> a rude critic,
>> > > >> I am just interested in consice and universal binary prorocol
>> > > >> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
>> > > >> valentin.kuliche...@gmail.com
>> > > >> >:
>> > > >>
>> > > >> > Ivan,
>> > > >> >
>> > > >> > KV view does work over the tuples. Nested objects and arbitrary
>> > > >> structures
>> > > >> > can be stored as blobs. So if you need a basic KV cache, you can
>> > > always
>> > > >> > create a table with two blob fields - one for key and one for
>> value
>> > -
>> > > >> and
>> > > >> > store anything there.
>> > > >> >
>> > > >> > -Val
>> > > >> >
>> > > >> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky <
>> ivanda...@gmail.com
>> > >
>> > > >> > wrote:
>> > > >> >
>> > > >> > > Val, am I right, that kv view over the tuples is just simple
>> > mapping
>> > > >> from
>> > > >> > > POJO to tuple? No collections, no nested objects? I have
>> discussed
>> > > >> this
>> > > >> > in
>> > > >> > > private with Igor and Pavel and they told me different info.
>> > > >> > >
>> > > >> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
>> > > >> > > valentin.kuliche...@gmail.com
>> > > >> > > >:
>> > > >> > >
>> > > >> > > > Ivan,
>> > > >> > > >
>> > > >> > > > I was answering your question about the KV API. The API I
>> > provided
>> > > >> has
>> > > >> > > been
>> > > >> > > > discussed and agreed upon. One of the goals of the protocol
>> is
&

Re: [DISCUSS] Confuse default inspections.

[Pavel Tupitsyn]

Agree with for both points

On Tue, Jul 20, 2021 at 3:14 PM Alexander Polovtcev 
wrote:

> this is a very welcome change for me
>
> On Tue, Jul 20, 2021 at 10:13 AM Ivan Pavlukhin 
> wrote:
>
> > + for both points.
> >
> > 2021-07-20 9:56 GMT+03:00, Ivan Daschinsky :
> > > Hi!
> > >
> > > Firstly, lets talk about interfaces.
> > >
> > > 1. First of all, do we have an automatic inspection for it? AFAIK we
> > don't
> > > 2. I am for consistency. At least for production code. Nothing worse is
> > > when someone mixes both approaches.
> > >
> > > About a prohibition of curly brackets around one line -- I am strongly
> > > against this rule, it should be removed. There were a few bugs that
> this
> > > rule caused.
> > >
> > > вт, 20 июл. 2021 г. в 09:23, Zhenya Stanilovsky
> >  > >>:
> > >
> > >>
> > >> Igniters, i understand that this is very long and fundamental story.
> but
> > >> … still want to rise up this discussion, we have 2 very strange
> > >> inspections:
> > >> *  «public» modifier in interface methods.
> > >> *  Illegal ‘{}’ for one line statement. — i found it harmful.
> > >> I don`t want to link additional discussion about pos. 2 i think that
> > harm
> > >> is obvious.
> > >> I suggest to get rid of them.
> > >>
> > >> what do you think ?
> > >>
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
> >
> > --
> >
> > Best regards,
> > Ivan Pavlukhin
> >
>
>
> --
> With regards,
> Aleksandr Polovtcev
>

Thin client data structures

[Pavel Tupitsyn]

Igniters,

I was asked a few times about Data Structures [1] in thin clients.

While there are plans to add them eventually, it is possible to implement
them on top of the existing thin client cache API.
I'd like to share a short blog post which demonstrates this [2].

[1] https://ignite.apache.org/features/datastructures.html
[2]
https://ptupitsyn.github.io/Ignite-Thin-Client-Distributed-Blocking-Queue/

Re: IEP-61 Transaction API desing for Ignite 3

[Pavel Tupitsyn]

> avoid any thread based control of transactions...
> single thread will be able to work with any amount of
> transactions at the same time

Yes, and one TX can be used by many threads, which is demonstrated
by testTxAsync.
I completely agree with this approach.

Ok, let's see what others say about "runInTransaction".






On Tue, Jul 13, 2021 at 6:51 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Pavel,
>
> "runInTransaction" is supposed to provide an "old-fashioned" way to write a
> transaction for easier migration.
>
> Manual enlisting of tables is required, because I strive to avoid any
> thread based control of transactions in Ignite 3.
>
> Actually, a single thread will be able to work with any amount of
> transactions at the same time.
>
> I would keep it for convenience, but let's see other opinions.
>
>
>
>
>
>
> вт, 13 июл. 2021 г. в 18:22, Pavel Tupitsyn :
>
> > Alexei,
> >
> > The API looks good to me, except "runInTransaction", which I find
> > confusing.
> >
> > It looks like every operation performed by the passed Consumer will be
> > automatically enlisted in a transaction,
> > but, looking at tests, "withTx" call is still required inside the
> Consumer.
> >
> > I don't think we need this method at all, it barely provides any
> > convenience but may confuse some users.
> >
> > On Tue, Jul 13, 2021 at 4:43 PM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I've prepared a PR implementing my vision of public transactions API.
> > >
> > > API is very simple and similar to Ignite 2, but has some differences.
> > >
> > > More details can be found here [1]
> > >
> > > Share your thoughts.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-15086
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: IEP-61 Transaction API desing for Ignite 3

[Pavel Tupitsyn]

Alexei,

The API looks good to me, except "runInTransaction", which I find confusing.

It looks like every operation performed by the passed Consumer will be
automatically enlisted in a transaction,
but, looking at tests, "withTx" call is still required inside the Consumer.

I don't think we need this method at all, it barely provides any
convenience but may confuse some users.

On Tue, Jul 13, 2021 at 4:43 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Folks,
>
> I've prepared a PR implementing my vision of public transactions API.
>
> API is very simple and similar to Ignite 2, but has some differences.
>
> More details can be found here [1]
>
> Share your thoughts.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-15086
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Val,

Ivan D has convinced me above to try the builder pattern, which is used by
many other drivers (Redis, Mongo, etc).
With IgniteClient class in ignite-client module:

Ignite client = IgniteClient.builder()
.setAddresses("127.0.0.1:10800")
.setTimeout(5000)
.build();

What do you think?

On Mon, Jul 12, 2021 at 10:57 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> If Ignition is the single entry point, it needs to have access to the
> server node instance, so it needs to depend on ignite-runner. Therefore,
> including Ignition in ignite-client means that ignite-client depends on
> ignite-runner, which we cannot have.
>
> -Val
>
> On Sat, Jul 10, 2021 at 4:17 AM Pavel Tupitsyn 
> wrote:
>
> > Val,
> >
> > My suggestion is to have Ignition class in ignite-client module.
> >
> > On Fri, Jul 9, 2021 at 11:01 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Pavel,
> > >
> > > Ivan actually brings a good point. While the client is in a separate
> > > module, Ignition (if we make it static) will have to depend on both
> > > ignite-client and ignite-runner, and we will have to ship it along with
> > the
> > > client. This indeed creates an uber-jar, so we can't really have a
> single
> > > entry point, unfortunately.
> > >
> > > I'm not sure what is the best way to proceed here. Let's think it over
> > and
> > > see if there are any suggestions.
> > >
> > > -Val
> > >
> > > On Fri, Jul 9, 2021 at 6:31 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > > why thin client should be in core module
> > > >
> > > > It will be in a separate module (ignite-client).
> > > > I was talking about "core library" as a primary set of modules that
> we
> > > > ship.
> > > > Integrations with 3rd party libraries and frameworks can be shipped
> as
> > > > extensions.
> > > >
> > > > Anyway, let's postpone the discussion of Rx and Kotlin.
> > > > The main goal right now is to implement the most basic Java thin
> > client.
> > > > CompletableFuture is the primary way to deliver async APIs in Ignite
> > 3.0,
> > > > other things can be added later.
> > > >
> > > > On Fri, Jul 9, 2021 at 3:37 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > > I don't think we need an explicit reactive API in the core
> library.
> > > > >
> > > > > Have you ever thought about why thin client should be in core
> module?
> > > Why
> > > > > we do the same thing as we did in ignite 2.x? In the days of cloud
> > > native
> > > > > we still think about large uber-jar with everything?
> > > > >
> > > > > > Same story with Kotlin, it works with CompletableFuture.
> > > > > Users don't want to code by theyselves, they want to use ready and
> > > > complete
> > > > > clients. Please, don't underestimate kotlin, kotlin coroutines and
> > > > reactive
> > > > > streams. They are all the first class citizens in spring 5 for 3
> > years
> > > > >
> > > > > пт, 9 июл. 2021 г., 14:43 Pavel Tupitsyn :
> > > > >
> > > > > > > You forget about reactive api
> > > > > >
> > > > > > I don't think we need an explicit reactive API in the core
> library.
> > > > > > Observable.fromFuture bridges async to Rx easily:
> > > > > >
> > > > > > Observable.fromFuture(client.putAsync(k, v)).flatMap(...)
> > > > > >
> > > > > > Same story with Kotlin, it works with CompletableFuture.
> > > > > >
> > > > > > On Fri, Jul 9, 2021 at 1:31 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > You forget about reactive api :)
> > > > > > >
> > > > > > > And whats a problem with discocerability?
> > > > > > >
> > > > > > > var syncApi = client.sync();
> > > > > > > syncApi.put(k, v);
> > > > > > >
> > > > > > > var rxApi = client.reactive();
> > > > > > > rxApi.put(k,v).flatMap(res -> );
> >

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Ivan D, ok, you have convinced me.
Builder pattern is popular for this sort of thing, and not only in the Java
world.

Let's see if there are other opinions.

On Sun, Jul 11, 2021 at 6:59 PM Ivan Daschinsky  wrote:

> The main reason is that immutable configurations nowadays is more
> preferable variant than mutable. For example, when you initialize client
> you must copy configuration in order to be consistent. Nobody nowadays read
> configuration from files as is, moreover -- the most microservices
> retrieves just simple string variables from env or from consul/etcd.
>
> вс, 11 июл. 2021 г., 17:48 Pavel Tupitsyn :
>
> > Serialization is just one example, and it does not have to be XML.
> > JSON, YAML, HOCON configs are widely used.
> >
> > Anyway, I see no reason for it NOT to be a POJO.
> > POJOs are ergonomic and work everywhere.
> >
> >
> > On Sun, Jul 11, 2021 at 8:24 AM Ivan Daschinsky 
> > wrote:
> >
> > > > But configuration should be a POJO - a class with setters and
> getters,
> > > nothing else.
> > >  Why it should? Why ClientConfiguration should be serializable? Who
> needs
> > > that? Xml configuration even in spring are not widely used for years :)
> > >
> > > вс, 11 июл. 2021 г., 00:57 Pavel Tupitsyn :
> > >
> > > > Ivan D,
> > > >
> > > > Let's compare WebClient approach with Ignite 2.x approach:
> > > >
> > > > // Ignite 2.x
> > > > var cfg = new ClientConfiguration()
> > > > .setAddresses("127.0.0.1:10800", "127.0.0.1:10801")
> > > > .setTimeout(4000);
> > > >
> > > > Ignite client = Ignition.startClient(cfg);
> > > >
> > > > // Ignite 3.x inspired by WebClient
> > > >
> > > > Ignite client = IgniteClient.builder()
> > > > .setAddresses("127.0.0.1:10800", "127.0.0.1:10801")
> > > > .setTimeout(4000)
> > > > .build();
> > > >
> > > >
> > > > To me, there is virtually no difference. Both are simple and
> > > > straightforward.
> > > > "specific static method within strange class with strange name"
> > (whatever
> > > > that means) can be applied equally to both.
> > > >
> > > > However, ClientConfiguration is a POJO with all the corresponding
> > > benefits,
> > > > like easy serialization.
> > > > WebClient.Builder is an interface that does not even provide a way to
> > get
> > > > the property value back.
> > > >
> > > >
> > > > To be clear, I don't have strong opinions on naming (it can be
> > > > Ignition.start or IgniteClient.connect or whatever).
> > > > But configuration should be a POJO - a class with setters and
> getters,
> > > > nothing else.
> > > >
> > > > On Sat, Jul 10, 2021 at 9:49 PM Ivan Daschinsky  >
> > > > wrote:
> > > >
> > > > > Pavel, I and Ivan P.  have already get examples of lettuce.io
> > > > >
> > > > > Another example is spring 5  reactive WebClient
> > > > > https://www.baeldung.com/spring-5-webclient
> > > > >
> > > > > сб, 10 июл. 2021 г., 19:50 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan D.,
> > > > > >
> > > > > > > simple and straightforward client builder
> > > > > > Can you please provide an example of that, a piece of code with
> > > > suggested
> > > > > > API usage? Ideally compared to the current 2.x approach.
> > > > > >
> > > > > > On Sat, Jul 10, 2021 at 3:20 PM Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > It is a quite questionable decision , as for me,  to have
> > specific
> > > > > static
> > > > > > > method within strange class with strange name (and it is a well
> > > known
> > > > > > > antipatter ) with chained twisted configuration class instead
> of
> > > just
> > > > > > > simple and straightforward client builder, isn't it? Name
> > > 'Ignition'
> > > > is
> > > > > > may
> > > > > > > be fun,

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Serialization is just one example, and it does not have to be XML.
JSON, YAML, HOCON configs are widely used.

Anyway, I see no reason for it NOT to be a POJO.
POJOs are ergonomic and work everywhere.


On Sun, Jul 11, 2021 at 8:24 AM Ivan Daschinsky  wrote:

> > But configuration should be a POJO - a class with setters and getters,
> nothing else.
>  Why it should? Why ClientConfiguration should be serializable? Who needs
> that? Xml configuration even in spring are not widely used for years :)
>
> вс, 11 июл. 2021 г., 00:57 Pavel Tupitsyn :
>
> > Ivan D,
> >
> > Let's compare WebClient approach with Ignite 2.x approach:
> >
> > // Ignite 2.x
> > var cfg = new ClientConfiguration()
> > .setAddresses("127.0.0.1:10800", "127.0.0.1:10801")
> > .setTimeout(4000);
> >
> > Ignite client = Ignition.startClient(cfg);
> >
> > // Ignite 3.x inspired by WebClient
> >
> > Ignite client = IgniteClient.builder()
> > .setAddresses("127.0.0.1:10800", "127.0.0.1:10801")
> > .setTimeout(4000)
> > .build();
> >
> >
> > To me, there is virtually no difference. Both are simple and
> > straightforward.
> > "specific static method within strange class with strange name" (whatever
> > that means) can be applied equally to both.
> >
> > However, ClientConfiguration is a POJO with all the corresponding
> benefits,
> > like easy serialization.
> > WebClient.Builder is an interface that does not even provide a way to get
> > the property value back.
> >
> >
> > To be clear, I don't have strong opinions on naming (it can be
> > Ignition.start or IgniteClient.connect or whatever).
> > But configuration should be a POJO - a class with setters and getters,
> > nothing else.
> >
> > On Sat, Jul 10, 2021 at 9:49 PM Ivan Daschinsky 
> > wrote:
> >
> > > Pavel, I and Ivan P.  have already get examples of lettuce.io
> > >
> > > Another example is spring 5  reactive WebClient
> > > https://www.baeldung.com/spring-5-webclient
> > >
> > > сб, 10 июл. 2021 г., 19:50 Pavel Tupitsyn :
> > >
> > > > Ivan D.,
> > > >
> > > > > simple and straightforward client builder
> > > > Can you please provide an example of that, a piece of code with
> > suggested
> > > > API usage? Ideally compared to the current 2.x approach.
> > > >
> > > > On Sat, Jul 10, 2021 at 3:20 PM Ivan Daschinsky  >
> > > > wrote:
> > > >
> > > > > It is a quite questionable decision , as for me,  to have specific
> > > static
> > > > > method within strange class with strange name (and it is a well
> known
> > > > > antipatter ) with chained twisted configuration class instead of
> just
> > > > > simple and straightforward client builder, isn't it? Name
> 'Ignition'
> > is
> > > > may
> > > > > be fun, but I cannot understand how it name connects to
> bootstraping
> > > > > ignite's client.
> > > > >
> > > > > сб, 10 июл. 2021 г., 14:17 Pavel Tupitsyn :
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > My suggestion is to have Ignition class in ignite-client module.
> > > > > >
> > > > > > On Fri, Jul 9, 2021 at 11:01 PM Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com> wrote:
> > > > > >
> > > > > > > Pavel,
> > > > > > >
> > > > > > > Ivan actually brings a good point. While the client is in a
> > > separate
> > > > > > > module, Ignition (if we make it static) will have to depend on
> > both
> > > > > > > ignite-client and ignite-runner, and we will have to ship it
> > along
> > > > with
> > > > > > the
> > > > > > > client. This indeed creates an uber-jar, so we can't really
> have
> > a
> > > > > single
> > > > > > > entry point, unfortunately.
> > > > > > >
> > > > > > > I'm not sure what is the best way to proceed here. Let's think
> it
> > > > over
> > > > > > and
> > > > > > > see if there are any suggest

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Ivan D,

Let's compare WebClient approach with Ignite 2.x approach:

// Ignite 2.x
var cfg = new ClientConfiguration()
.setAddresses("127.0.0.1:10800", "127.0.0.1:10801")
.setTimeout(4000);

Ignite client = Ignition.startClient(cfg);

// Ignite 3.x inspired by WebClient

Ignite client = IgniteClient.builder()
.setAddresses("127.0.0.1:10800", "127.0.0.1:10801")
.setTimeout(4000)
.build();


To me, there is virtually no difference. Both are simple and
straightforward.
"specific static method within strange class with strange name" (whatever
that means) can be applied equally to both.

However, ClientConfiguration is a POJO with all the corresponding benefits,
like easy serialization.
WebClient.Builder is an interface that does not even provide a way to get
the property value back.


To be clear, I don't have strong opinions on naming (it can be
Ignition.start or IgniteClient.connect or whatever).
But configuration should be a POJO - a class with setters and getters,
nothing else.

On Sat, Jul 10, 2021 at 9:49 PM Ivan Daschinsky  wrote:

> Pavel, I and Ivan P.  have already get examples of lettuce.io
>
> Another example is spring 5  reactive WebClient
> https://www.baeldung.com/spring-5-webclient
>
> сб, 10 июл. 2021 г., 19:50 Pavel Tupitsyn :
>
> > Ivan D.,
> >
> > > simple and straightforward client builder
> > Can you please provide an example of that, a piece of code with suggested
> > API usage? Ideally compared to the current 2.x approach.
> >
> > On Sat, Jul 10, 2021 at 3:20 PM Ivan Daschinsky 
> > wrote:
> >
> > > It is a quite questionable decision , as for me,  to have specific
> static
> > > method within strange class with strange name (and it is a well known
> > > antipatter ) with chained twisted configuration class instead of just
> > > simple and straightforward client builder, isn't it? Name 'Ignition' is
> > may
> > > be fun, but I cannot understand how it name connects to bootstraping
> > > ignite's client.
> > >
> > > сб, 10 июл. 2021 г., 14:17 Pavel Tupitsyn :
> > >
> > > > Val,
> > > >
> > > > My suggestion is to have Ignition class in ignite-client module.
> > > >
> > > > On Fri, Jul 9, 2021 at 11:01 PM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Pavel,
> > > > >
> > > > > Ivan actually brings a good point. While the client is in a
> separate
> > > > > module, Ignition (if we make it static) will have to depend on both
> > > > > ignite-client and ignite-runner, and we will have to ship it along
> > with
> > > > the
> > > > > client. This indeed creates an uber-jar, so we can't really have a
> > > single
> > > > > entry point, unfortunately.
> > > > >
> > > > > I'm not sure what is the best way to proceed here. Let's think it
> > over
> > > > and
> > > > > see if there are any suggestions.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Jul 9, 2021 at 6:31 AM Pavel Tupitsyn <
> ptupit...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > > why thin client should be in core module
> > > > > >
> > > > > > It will be in a separate module (ignite-client).
> > > > > > I was talking about "core library" as a primary set of modules
> that
> > > we
> > > > > > ship.
> > > > > > Integrations with 3rd party libraries and frameworks can be
> shipped
> > > as
> > > > > > extensions.
> > > > > >
> > > > > > Anyway, let's postpone the discussion of Rx and Kotlin.
> > > > > > The main goal right now is to implement the most basic Java thin
> > > > client.
> > > > > > CompletableFuture is the primary way to deliver async APIs in
> > Ignite
> > > > 3.0,
> > > > > > other things can be added later.
> > > > > >
> > > > > > On Fri, Jul 9, 2021 at 3:37 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > > I don't think we need an explicit reactive API in the core
> > > library.
&g

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Ivan D.,

> simple and straightforward client builder
Can you please provide an example of that, a piece of code with suggested
API usage? Ideally compared to the current 2.x approach.

On Sat, Jul 10, 2021 at 3:20 PM Ivan Daschinsky  wrote:

> It is a quite questionable decision , as for me,  to have specific static
> method within strange class with strange name (and it is a well known
> antipatter ) with chained twisted configuration class instead of just
> simple and straightforward client builder, isn't it? Name 'Ignition' is may
> be fun, but I cannot understand how it name connects to bootstraping
> ignite's client.
>
> сб, 10 июл. 2021 г., 14:17 Pavel Tupitsyn :
>
> > Val,
> >
> > My suggestion is to have Ignition class in ignite-client module.
> >
> > On Fri, Jul 9, 2021 at 11:01 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Pavel,
> > >
> > > Ivan actually brings a good point. While the client is in a separate
> > > module, Ignition (if we make it static) will have to depend on both
> > > ignite-client and ignite-runner, and we will have to ship it along with
> > the
> > > client. This indeed creates an uber-jar, so we can't really have a
> single
> > > entry point, unfortunately.
> > >
> > > I'm not sure what is the best way to proceed here. Let's think it over
> > and
> > > see if there are any suggestions.
> > >
> > > -Val
> > >
> > > On Fri, Jul 9, 2021 at 6:31 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > > why thin client should be in core module
> > > >
> > > > It will be in a separate module (ignite-client).
> > > > I was talking about "core library" as a primary set of modules that
> we
> > > > ship.
> > > > Integrations with 3rd party libraries and frameworks can be shipped
> as
> > > > extensions.
> > > >
> > > > Anyway, let's postpone the discussion of Rx and Kotlin.
> > > > The main goal right now is to implement the most basic Java thin
> > client.
> > > > CompletableFuture is the primary way to deliver async APIs in Ignite
> > 3.0,
> > > > other things can be added later.
> > > >
> > > > On Fri, Jul 9, 2021 at 3:37 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > > I don't think we need an explicit reactive API in the core
> library.
> > > > >
> > > > > Have you ever thought about why thin client should be in core
> module?
> > > Why
> > > > > we do the same thing as we did in ignite 2.x? In the days of cloud
> > > native
> > > > > we still think about large uber-jar with everything?
> > > > >
> > > > > > Same story with Kotlin, it works with CompletableFuture.
> > > > > Users don't want to code by theyselves, they want to use ready and
> > > > complete
> > > > > clients. Please, don't underestimate kotlin, kotlin coroutines and
> > > > reactive
> > > > > streams. They are all the first class citizens in spring 5 for 3
> > years
> > > > >
> > > > > пт, 9 июл. 2021 г., 14:43 Pavel Tupitsyn :
> > > > >
> > > > > > > You forget about reactive api
> > > > > >
> > > > > > I don't think we need an explicit reactive API in the core
> library.
> > > > > > Observable.fromFuture bridges async to Rx easily:
> > > > > >
> > > > > > Observable.fromFuture(client.putAsync(k, v)).flatMap(...)
> > > > > >
> > > > > > Same story with Kotlin, it works with CompletableFuture.
> > > > > >
> > > > > > On Fri, Jul 9, 2021 at 1:31 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > You forget about reactive api :)
> > > > > > >
> > > > > > > And whats a problem with discocerability?
> > > > > > >
> > > > > > > var syncApi = client.sync();
> > > > > > > syncApi.put(k, v);
> > > > > > >
> > > > > > > var rxApi = client.reactive();
> > > > > > > rxApi.put(k,v).flatMap(res -> );
> > > > > > >
> > > > > > > An

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Val,

My suggestion is to have Ignition class in ignite-client module.

On Fri, Jul 9, 2021 at 11:01 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> Ivan actually brings a good point. While the client is in a separate
> module, Ignition (if we make it static) will have to depend on both
> ignite-client and ignite-runner, and we will have to ship it along with the
> client. This indeed creates an uber-jar, so we can't really have a single
> entry point, unfortunately.
>
> I'm not sure what is the best way to proceed here. Let's think it over and
> see if there are any suggestions.
>
> -Val
>
> On Fri, Jul 9, 2021 at 6:31 AM Pavel Tupitsyn 
> wrote:
>
> > > why thin client should be in core module
> >
> > It will be in a separate module (ignite-client).
> > I was talking about "core library" as a primary set of modules that we
> > ship.
> > Integrations with 3rd party libraries and frameworks can be shipped as
> > extensions.
> >
> > Anyway, let's postpone the discussion of Rx and Kotlin.
> > The main goal right now is to implement the most basic Java thin client.
> > CompletableFuture is the primary way to deliver async APIs in Ignite 3.0,
> > other things can be added later.
> >
> > On Fri, Jul 9, 2021 at 3:37 PM Ivan Daschinsky 
> > wrote:
> >
> > > > I don't think we need an explicit reactive API in the core library.
> > >
> > > Have you ever thought about why thin client should be in core module?
> Why
> > > we do the same thing as we did in ignite 2.x? In the days of cloud
> native
> > > we still think about large uber-jar with everything?
> > >
> > > > Same story with Kotlin, it works with CompletableFuture.
> > > Users don't want to code by theyselves, they want to use ready and
> > complete
> > > clients. Please, don't underestimate kotlin, kotlin coroutines and
> > reactive
> > > streams. They are all the first class citizens in spring 5 for 3 years
> > >
> > > пт, 9 июл. 2021 г., 14:43 Pavel Tupitsyn :
> > >
> > > > > You forget about reactive api
> > > >
> > > > I don't think we need an explicit reactive API in the core library.
> > > > Observable.fromFuture bridges async to Rx easily:
> > > >
> > > > Observable.fromFuture(client.putAsync(k, v)).flatMap(...)
> > > >
> > > > Same story with Kotlin, it works with CompletableFuture.
> > > >
> > > > On Fri, Jul 9, 2021 at 1:31 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > You forget about reactive api :)
> > > > >
> > > > > And whats a problem with discocerability?
> > > > >
> > > > > var syncApi = client.sync();
> > > > > syncApi.put(k, v);
> > > > >
> > > > > var rxApi = client.reactive();
> > > > > rxApi.put(k,v).flatMap(res -> );
> > > > >
> > > > > And sync, async and reactive is not enough, it is good idea to
> > support
> > > > > kotlin coroutines also :)
> > > > >
> > > > > пт, 9 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan D.,
> > > > > >
> > > > > > > container of properties
> > > > > >
> > > > > > What is a container of properties?
> > > > > > As a user, I want a simple way to start a client and perform
> > > > operations.
> > > > > >
> > > > > > I don't want anything confusing and complicated like Netty
> > Bootstrap.
> > > > > There
> > > > > > might be a reason for Netty to be this way - it is a low-level
> > > library.
> > > > > But
> > > > > > Ignite is not.
> > > > > >
> > > > > >
> > > > > > > separate facades for sync, async
> > > > > >
> > > > > > Strongly disagree with this idea. It hurts API discoverability.
> > > > > > As a user, in my IDE I type "igniteTable.get" and see a list of
> > > > > suggestions
> > > > > > like get, getAsync, getAndPut, getAndPutAsync.
> > > > > > I don't want to have a separate interface and a separate variable
> > to
> > > > deal
> > > > > > with sync and async methods.
&

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

> why thin client should be in core module

It will be in a separate module (ignite-client).
I was talking about "core library" as a primary set of modules that we ship.
Integrations with 3rd party libraries and frameworks can be shipped as
extensions.

Anyway, let's postpone the discussion of Rx and Kotlin.
The main goal right now is to implement the most basic Java thin client.
CompletableFuture is the primary way to deliver async APIs in Ignite 3.0,
other things can be added later.

On Fri, Jul 9, 2021 at 3:37 PM Ivan Daschinsky  wrote:

> > I don't think we need an explicit reactive API in the core library.
>
> Have you ever thought about why thin client should be in core module? Why
> we do the same thing as we did in ignite 2.x? In the days of cloud native
> we still think about large uber-jar with everything?
>
> > Same story with Kotlin, it works with CompletableFuture.
> Users don't want to code by theyselves, they want to use ready and complete
> clients. Please, don't underestimate kotlin, kotlin coroutines and reactive
> streams. They are all the first class citizens in spring 5 for 3 years
>
> пт, 9 июл. 2021 г., 14:43 Pavel Tupitsyn :
>
> > > You forget about reactive api
> >
> > I don't think we need an explicit reactive API in the core library.
> > Observable.fromFuture bridges async to Rx easily:
> >
> > Observable.fromFuture(client.putAsync(k, v)).flatMap(...)
> >
> > Same story with Kotlin, it works with CompletableFuture.
> >
> > On Fri, Jul 9, 2021 at 1:31 PM Ivan Daschinsky 
> > wrote:
> >
> > > You forget about reactive api :)
> > >
> > > And whats a problem with discocerability?
> > >
> > > var syncApi = client.sync();
> > > syncApi.put(k, v);
> > >
> > > var rxApi = client.reactive();
> > > rxApi.put(k,v).flatMap(res -> );
> > >
> > > And sync, async and reactive is not enough, it is good idea to support
> > > kotlin coroutines also :)
> > >
> > > пт, 9 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > >
> > > > Ivan D.,
> > > >
> > > > > container of properties
> > > >
> > > > What is a container of properties?
> > > > As a user, I want a simple way to start a client and perform
> > operations.
> > > >
> > > > I don't want anything confusing and complicated like Netty Bootstrap.
> > > There
> > > > might be a reason for Netty to be this way - it is a low-level
> library.
> > > But
> > > > Ignite is not.
> > > >
> > > >
> > > > > separate facades for sync, async
> > > >
> > > > Strongly disagree with this idea. It hurts API discoverability.
> > > > As a user, in my IDE I type "igniteTable.get" and see a list of
> > > suggestions
> > > > like get, getAsync, getAndPut, getAndPutAsync.
> > > > I don't want to have a separate interface and a separate variable to
> > deal
> > > > with sync and async methods.
> > > >
> > > > Not sure what's the problem with documentation - can you elaborate
> > > please?
> > > >
> > > > On Fri, Jul 9, 2021 at 12:51 PM Ivan Daschinsky  >
> > > > wrote:
> > > >
> > > > > Pavel, actually I suggests to separate container of
> properties(client
> > > in
> > > > > lettuce) and actual connection or connections (stateful connection
> in
> > > > > lettuce). Actual connection initialization could be sync or async,
> it
> > > > > doesn't matter. It can be Ignition#startClient or
> > > > > Ignition#startClientAsync, but I'd prefer lettuce approach
> > > > >
> > > > >
> > > > > Also, it would be great to have separate facades for sync, async
> and
> > > > > reactive api. Mixing all of them in one interface is a
> documentation
> > > > > nightmare.
> > > > >
> > > > > пт, 9 июл. 2021 г., 11:55 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan P., Ivan D.,
> > > > > >
> > > > > > I don't think it makes sense to separate IgniteConnection and
> > > > > IgniteClient
> > > > > > like Lettuce does,
> > > > > > because IgniteClient will maintain connections to multiple server
> > > nodes
> > > > > > automatically,
> > > > > > and the numbe

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

> You forget about reactive api

I don't think we need an explicit reactive API in the core library.
Observable.fromFuture bridges async to Rx easily:

Observable.fromFuture(client.putAsync(k, v)).flatMap(...)

Same story with Kotlin, it works with CompletableFuture.

On Fri, Jul 9, 2021 at 1:31 PM Ivan Daschinsky  wrote:

> You forget about reactive api :)
>
> And whats a problem with discocerability?
>
> var syncApi = client.sync();
> syncApi.put(k, v);
>
> var rxApi = client.reactive();
> rxApi.put(k,v).flatMap(res -> );
>
> And sync, async and reactive is not enough, it is good idea to support
> kotlin coroutines also :)
>
> пт, 9 июл. 2021 г., 13:26 Pavel Tupitsyn :
>
> > Ivan D.,
> >
> > > container of properties
> >
> > What is a container of properties?
> > As a user, I want a simple way to start a client and perform operations.
> >
> > I don't want anything confusing and complicated like Netty Bootstrap.
> There
> > might be a reason for Netty to be this way - it is a low-level library.
> But
> > Ignite is not.
> >
> >
> > > separate facades for sync, async
> >
> > Strongly disagree with this idea. It hurts API discoverability.
> > As a user, in my IDE I type "igniteTable.get" and see a list of
> suggestions
> > like get, getAsync, getAndPut, getAndPutAsync.
> > I don't want to have a separate interface and a separate variable to deal
> > with sync and async methods.
> >
> > Not sure what's the problem with documentation - can you elaborate
> please?
> >
> > On Fri, Jul 9, 2021 at 12:51 PM Ivan Daschinsky 
> > wrote:
> >
> > > Pavel, actually I suggests to separate container of properties(client
> in
> > > lettuce) and actual connection or connections (stateful connection in
> > > lettuce). Actual connection initialization could be sync or async, it
> > > doesn't matter. It can be Ignition#startClient or
> > > Ignition#startClientAsync, but I'd prefer lettuce approach
> > >
> > >
> > > Also, it would be great to have separate facades for sync, async and
> > > reactive api. Mixing all of them in one interface is a documentation
> > > nightmare.
> > >
> > > пт, 9 июл. 2021 г., 11:55 Pavel Tupitsyn :
> > >
> > > > Ivan P., Ivan D.,
> > > >
> > > > I don't think it makes sense to separate IgniteConnection and
> > > IgniteClient
> > > > like Lettuce does,
> > > > because IgniteClient will maintain connections to multiple server
> nodes
> > > > automatically,
> > > > and the number of connections can grow and shrink dynamically.
> > > >
> > > > This is required to support dynamic clusters together with partition
> > > > awareness.
> > > >
> > > >
> > > > > Why not to make async variant of connection
> > > >
> > > > Ignite API will (eventually) have both sync and async variants of
> every
> > > > method, where applicable,
> > > > including the method that connects the client to the cluster.
> > > >
> > > > On Fri, Jul 9, 2021 at 9:55 AM Ivan Pavlukhin 
> > > wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > > Ignition IS the entry point to Ignite, so I'm not sure I got your
> > > point
> > > > > :)
> > > > > > Either way, please feel free to give your suggestions for an
> > > > alternative
> > > > > name if you have any.
> > > > >
> > > > > Well, it is not only about naming but it is also about code
> > > > > organization. Ivan D. already referenced to alternative API styles
> (I
> > > > > suppose [1] describes the idea).
> > > > >
> > > > > My main points are:
> > > > > 1. Ignite 3 is a great opportunity to make things better.
> > > > > 2. Using (or reusing) confusing names and entities should be
> avoided.
> > > > >
> > > > > Another rather straightforward startup/bootstrap approach is used
> in
> > > > > Netty [2]. For Ignite I would spell it like IgniteServer.Bootstrap
> > and
> > > > > IgniteClient.Bootstrap.
> > > > >
> > > > > Also I suppose that thin client API is more important because much
> > > > > more users will use it. I hope that a lot of Community mem

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Ivan D.,

> container of properties

What is a container of properties?
As a user, I want a simple way to start a client and perform operations.

I don't want anything confusing and complicated like Netty Bootstrap. There
might be a reason for Netty to be this way - it is a low-level library. But
Ignite is not.


> separate facades for sync, async

Strongly disagree with this idea. It hurts API discoverability.
As a user, in my IDE I type "igniteTable.get" and see a list of suggestions
like get, getAsync, getAndPut, getAndPutAsync.
I don't want to have a separate interface and a separate variable to deal
with sync and async methods.

Not sure what's the problem with documentation - can you elaborate please?

On Fri, Jul 9, 2021 at 12:51 PM Ivan Daschinsky  wrote:

> Pavel, actually I suggests to separate container of properties(client in
> lettuce) and actual connection or connections (stateful connection in
> lettuce). Actual connection initialization could be sync or async, it
> doesn't matter. It can be Ignition#startClient or
> Ignition#startClientAsync, but I'd prefer lettuce approach
>
>
> Also, it would be great to have separate facades for sync, async and
> reactive api. Mixing all of them in one interface is a documentation
> nightmare.
>
> пт, 9 июл. 2021 г., 11:55 Pavel Tupitsyn :
>
> > Ivan P., Ivan D.,
> >
> > I don't think it makes sense to separate IgniteConnection and
> IgniteClient
> > like Lettuce does,
> > because IgniteClient will maintain connections to multiple server nodes
> > automatically,
> > and the number of connections can grow and shrink dynamically.
> >
> > This is required to support dynamic clusters together with partition
> > awareness.
> >
> >
> > > Why not to make async variant of connection
> >
> > Ignite API will (eventually) have both sync and async variants of every
> > method, where applicable,
> > including the method that connects the client to the cluster.
> >
> > On Fri, Jul 9, 2021 at 9:55 AM Ivan Pavlukhin 
> wrote:
> >
> > > Val,
> > >
> > > > Ignition IS the entry point to Ignite, so I'm not sure I got your
> point
> > > :)
> > > > Either way, please feel free to give your suggestions for an
> > alternative
> > > name if you have any.
> > >
> > > Well, it is not only about naming but it is also about code
> > > organization. Ivan D. already referenced to alternative API styles (I
> > > suppose [1] describes the idea).
> > >
> > > My main points are:
> > > 1. Ignite 3 is a great opportunity to make things better.
> > > 2. Using (or reusing) confusing names and entities should be avoided.
> > >
> > > Another rather straightforward startup/bootstrap approach is used in
> > > Netty [2]. For Ignite I would spell it like IgniteServer.Bootstrap and
> > > IgniteClient.Bootstrap.
> > >
> > > Also I suppose that thin client API is more important because much
> > > more users will use it. I hope that a lot of Community members will
> > > share their ideas.
> > >
> > > [1] https://www.baeldung.com/java-redis-lettuce
> > > [2] https://netty.io/4.0/api/io/netty/bootstrap/ServerBootstrap.html
> > >
> > > 2021-07-09 1:41 GMT+03:00, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > > > Ivan,
> > > >
> > > > I've seen the link, but I still don't understand what exactly you
> > propose
> > > > to change in the current API, and what is your concern. Could you
> > please
> > > > clarify? How you think Ignite API should look like?
> > > >
> > > > -Val
> > > >
> > > > On Thu, Jul 8, 2021 at 2:18 PM Ivan Daschinsky 
> > > wrote:
> > > >
> > > >> Val, I have already gave examples -- lettuce, a very performant and
> > > >> modern
> > > >> redis java client
> > > >>
> > > >> I can duplicate links again
> > > >>
> https://lettuce.io/core/release/api/io/lettuce/core/RedisClient.html
> > > >>
> > > >>
> > >
> >
> https://lettuce.io/core/release/api/io/lettuce/core/api/StatefulRedisConnection.html
> > > >> https://www.baeldung.com/java-redis-lettuce
> > > >>
> > > >> чт, 8 июл. 2021 г., 23:47 Valentin Kulichenko <
> > > >> valentin.kuliche...@gmail.com
> > > >> >:
> > > >>
> > > >

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Ivan P., Ivan D.,

I don't think it makes sense to separate IgniteConnection and IgniteClient
like Lettuce does,
because IgniteClient will maintain connections to multiple server nodes
automatically,
and the number of connections can grow and shrink dynamically.

This is required to support dynamic clusters together with partition
awareness.


> Why not to make async variant of connection

Ignite API will (eventually) have both sync and async variants of every
method, where applicable,
including the method that connects the client to the cluster.

On Fri, Jul 9, 2021 at 9:55 AM Ivan Pavlukhin  wrote:

> Val,
>
> > Ignition IS the entry point to Ignite, so I'm not sure I got your point
> :)
> > Either way, please feel free to give your suggestions for an alternative
> name if you have any.
>
> Well, it is not only about naming but it is also about code
> organization. Ivan D. already referenced to alternative API styles (I
> suppose [1] describes the idea).
>
> My main points are:
> 1. Ignite 3 is a great opportunity to make things better.
> 2. Using (or reusing) confusing names and entities should be avoided.
>
> Another rather straightforward startup/bootstrap approach is used in
> Netty [2]. For Ignite I would spell it like IgniteServer.Bootstrap and
> IgniteClient.Bootstrap.
>
> Also I suppose that thin client API is more important because much
> more users will use it. I hope that a lot of Community members will
> share their ideas.
>
> [1] https://www.baeldung.com/java-redis-lettuce
> [2] https://netty.io/4.0/api/io/netty/bootstrap/ServerBootstrap.html
>
> 2021-07-09 1:41 GMT+03:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> > Ivan,
> >
> > I've seen the link, but I still don't understand what exactly you propose
> > to change in the current API, and what is your concern. Could you please
> > clarify? How you think Ignite API should look like?
> >
> > -Val
> >
> > On Thu, Jul 8, 2021 at 2:18 PM Ivan Daschinsky 
> wrote:
> >
> >> Val, I have already gave examples -- lettuce, a very performant and
> >> modern
> >> redis java client
> >>
> >> I can duplicate links again
> >> https://lettuce.io/core/release/api/io/lettuce/core/RedisClient.html
> >>
> >>
> https://lettuce.io/core/release/api/io/lettuce/core/api/StatefulRedisConnection.html
> >> https://www.baeldung.com/java-redis-lettuce
> >>
> >> чт, 8 июл. 2021 г., 23:47 Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com
> >> >:
> >>
> >> > Ivan,
> >> >
> >> > Can you please clarify what you mean by "separate creation of client
> >> > and
> >> > connection"? Can you give an example?
> >> >
> >> > -Val
> >> >
> >> > On Thu, Jul 8, 2021 at 12:53 PM Ivan Daschinsky 
> >> > wrote:
> >> >
> >> > > I'm sorry, but why we didn't consider to separate creation of Client
> >> and
> >> > > connection? Why not to make async variant of connection? See for
> >> example
> >> > > [1]
> >> > > [1] --- https://lettuce.io/core/release/api/index.html
> >> > >
> >> > >
> >> > > чт, 8 июл. 2021 г., 09:50 Pavel Tupitsyn :
> >> > >
> >> > > > Val,
> >> > > >
> >> > > > So the plan is:
> >> > > >
> >> > > > - Remove Ignition#start from the public API
> >> > > > - Make Ignition a class, not an interface
> >> > > > - Add static Ignition#startClient
> >> > > >
> >> > > > Sounds good?
> >> > > >
> >> > > > On Thu, Jul 8, 2021 at 6:13 AM Valentin Kulichenko <
> >> > > > valentin.kuliche...@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Ivan,
> >> > > > >
> >> > > > > Ignition IS the entry point to Ignite, so I'm not sure I got
> your
> >> > point
> >> > > > :)
> >> > > > > Where is the contradiction?
> >> > > > >
> >> > > > > Either way, please feel free to give your suggestions for an
> >> > > alternative
> >> > > > > name if you have any.
> >> > > > >
> >> > > > > -Val
> >> > > > >
> >> > > > > On Wed, Jul 7, 2021 at 7:56 PM Ivan Pavlukhina <

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Val,

So the plan is:

- Remove Ignition#start from the public API
- Make Ignition a class, not an interface
- Add static Ignition#startClient

Sounds good?

On Thu, Jul 8, 2021 at 6:13 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Ivan,
>
> Ignition IS the entry point to Ignite, so I'm not sure I got your point :)
> Where is the contradiction?
>
> Either way, please feel free to give your suggestions for an alternative
> name if you have any.
>
> -Val
>
> On Wed, Jul 7, 2021 at 7:56 PM Ivan Pavlukhina 
> wrote:
>
> > A side note. Actually “Ignition” naming always confused me. I think about
> > it as some fancy named API entry point for Ignite. Perhaps it is a good
> > moment to revisit naming.
> >
> > > On 8 Jul 2021, at 07:09, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> > >
> > > Hi Pavel,
> > >
> > > I don't think we will need the pure embedded mode, but we still need to
> > be
> > > able to access the API from compute and services. That said, there are
> > two
> > > usages of the 'Ignite' API:
> > >
> > >   1. Remote, via the binary protocol.
> > >   2. Local - needed for compute and services. (This is how it works
> now.)
> > >
> > > I believe that the API should be the same, and there should be a
> unified
> > > access point. Ignition seems to be a good candidate for this.
> > >
> > > Ignition#start should eventually be removed from the public API. It is
> > > currently there only because we don't have the thin client yet.
> > >
> > > -Val
> > >
> > >> On Wed, Jul 7, 2021 at 5:47 AM Pavel Tupitsyn 
> > wrote:
> > >>
> > >> Igniters,
> > >>
> > >> I have a few questions regarding server node startup and thin clients.
> > >>
> > >> State of things:
> > >> - Server nodes will be started with 'ignite run' from CLI [1]
> > >> - ignite-api module represents our public API
> > >> - ignite-api has Ignition interface to start server nodes
> > >>
> > >> Questions:
> > >> - What's the idea behind Ignition interface in the public API? Are we
> > going
> > >> to have an "embedded mode" where servers can be started from code? I
> > >> thought this was not planned.
> > >> - How are users supposed to retrieve an instance of the Ignition
> > interface?
> > >> - Are there any plans to start thin clients from Ignition interface,
> or
> > >> should we have a separate way of doing this?
> > >>
> > >>
> > >> [1]
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > >>
> >
>

Ignite 3.0 Ignition API, node startup, and thin client startup

[Pavel Tupitsyn]

Igniters,

I have a few questions regarding server node startup and thin clients.

State of things:
- Server nodes will be started with 'ignite run' from CLI [1]
- ignite-api module represents our public API
- ignite-api has Ignition interface to start server nodes

Questions:
- What's the idea behind Ignition interface in the public API? Are we going
to have an "embedded mode" where servers can be started from code? I
thought this was not planned.
- How are users supposed to retrieve an instance of the Ignition interface?
- Are there any plans to start thin clients from Ignition interface, or
should we have a separate way of doing this?


[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Thanks everyone, I've moved the proposal to Active status.
Working on the implementation.

On Tue, Jul 6, 2021 at 10:31 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> The proposal looks good to me.
>
> -Val
>
> On Tue, Jul 6, 2021 at 2:24 AM Ivan Daschinsky 
> wrote:
>
> > I suppose, that general idea is great. Some details are missing, but I
> > suppose during implementation of clients we will add more details and
> > redefine some parts.
> >
> > вт, 6 июл. 2021 г., 09:59 Pavel Tupitsyn :
> >
> > > Ivan, Val, and others - are there any open objections or questions?
> > > Can we accept the proposal?
> > >
> > > On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I've updated the IEP to support "Live Schema" [1] from IEP-54.
> > > > Some operations now have schemaless variants, where tuples are
> > serialized
> > > > as maps (String -> val).
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
> > > >
> > > > On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > >> Val, my understanding about it was exactly the same as yours, but,
> > > again,
> > > >> I
> > > >> heard a different opinion.
> > > >>
> > > >> But nevertheless, binary protocol should not be about objects,
> records
> > > aka
> > > >> tuples are the best varii, simple and powerful.
> > > >>
> > > >> As for me, I want to take part in implementing python and golang
> thin
> > > >> clients for ignite 3, so consider my remarks using this info. I am
> not
> > > >> just
> > > >> a rude critic,
> > > >> I am just interested in consice and universal binary prorocol
> > > >> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
> > > >> valentin.kuliche...@gmail.com
> > > >> >:
> > > >>
> > > >> > Ivan,
> > > >> >
> > > >> > KV view does work over the tuples. Nested objects and arbitrary
> > > >> structures
> > > >> > can be stored as blobs. So if you need a basic KV cache, you can
> > > always
> > > >> > create a table with two blob fields - one for key and one for
> value
> > -
> > > >> and
> > > >> > store anything there.
> > > >> >
> > > >> > -Val
> > > >> >
> > > >> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Val, am I right, that kv view over the tuples is just simple
> > mapping
> > > >> from
> > > >> > > POJO to tuple? No collections, no nested objects? I have
> discussed
> > > >> this
> > > >> > in
> > > >> > > private with Igor and Pavel and they told me different info.
> > > >> > >
> > > >> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> > > >> > > valentin.kuliche...@gmail.com
> > > >> > > >:
> > > >> > >
> > > >> > > > Ivan,
> > > >> > > >
> > > >> > > > I was answering your question about the KV API. The API I
> > provided
> > > >> has
> > > >> > > been
> > > >> > > > discussed and agreed upon. One of the goals of the protocol is
> > to
> > > >> > > implement
> > > >> > > > this API, so it should give you a clear idea of what we're
> > looking
> > > >> for.
> > > >> > > >
> > > >> > > > Of course, I agree with you that the protocol should be simple
> > and
> > > >> > > flexible
> > > >> > > > enough to allow other implementations for different languages
> > and
> > > >> > > > platforms.
> > > >> > > >
> > > >> > > > -Val
> > > >> > > >
> > > >> > &

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Pavel Tupitsyn]

Ivan,

Nothing wrong except for performance concerns.
The following code looks up the column by name twice:

if (!tuple.isNull("weight"))
   doSomething(tuple.floatValue("weight"))

Whereas in other languages you could do it in one shot:

if (tuple.TryGetFloatValue("weight", out var weight))
doSomething(weight)

or Option weight = tuple.floatValue("weight") and so on.

On Tue, Jul 6, 2021 at 9:58 AM Ivan Daschinsky  wrote:

> Sorry, but what is wrong with simple method isNull()
>
> вт, 6 июл. 2021 г., 09:55 Pavel Tupitsyn :
>
> > Val,
> >
> > > I don't think there is a significantly better way
> > > of doing this in Java.
> >
> > Yep looks like there is no way to return two values without boxing.
> > No ref, no out, no value types.
> >
> > > Schema already provides this information, doesn't it?
> >
> > It does, though we don't have an agreement on how to expose this on
> public
> > API yet,
> > or do we?
> >
> > On Tue, Jul 6, 2021 at 12:44 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Pavel,
> > >
> > > That's a good point, but I don't think there is a significantly better
> > way
> > > of doing this in Java.
> > >
> > > There should be a way to check if a field is nullable or not though.
> > Schema
> > > already provides this information, doesn't it?
> > >
> > > -Val
> > >
> > > On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > Looks like Tuple API has no efficient way to tell if a value for a
> > > nullable
> > > > column of primitive type is null.
> > > >
> > > > - Tuple#intValue() will return 0 when the actual value is null => not
> > > clear
> > > > if 0 is 0 or null.
> > > > - Tuple#value() works, but is more expensive due to boxing and type
> > > lookup.
> > > >
> > > > Any ideas on how to improve this?
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan, Val, and others - are there any open objections or questions?
Can we accept the proposal?

On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn  wrote:

> Igniters,
>
> I've updated the IEP to support "Live Schema" [1] from IEP-54.
> Some operations now have schemaless variants, where tuples are serialized
> as maps (String -> val).
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
>
> On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky 
> wrote:
>
>> Val, my understanding about it was exactly the same as yours, but, again,
>> I
>> heard a different opinion.
>>
>> But nevertheless, binary protocol should not be about objects, records aka
>> tuples are the best varii, simple and powerful.
>>
>> As for me, I want to take part in implementing python and golang thin
>> clients for ignite 3, so consider my remarks using this info. I am not
>> just
>> a rude critic,
>> I am just interested in consice and universal binary prorocol
>> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
>> valentin.kuliche...@gmail.com
>> >:
>>
>> > Ivan,
>> >
>> > KV view does work over the tuples. Nested objects and arbitrary
>> structures
>> > can be stored as blobs. So if you need a basic KV cache, you can always
>> > create a table with two blob fields - one for key and one for value -
>> and
>> > store anything there.
>> >
>> > -Val
>> >
>> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky 
>> > wrote:
>> >
>> > > Val, am I right, that kv view over the tuples is just simple mapping
>> from
>> > > POJO to tuple? No collections, no nested objects? I have discussed
>> this
>> > in
>> > > private with Igor and Pavel and they told me different info.
>> > >
>> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
>> > > valentin.kuliche...@gmail.com
>> > > >:
>> > >
>> > > > Ivan,
>> > > >
>> > > > I was answering your question about the KV API. The API I provided
>> has
>> > > been
>> > > > discussed and agreed upon. One of the goals of the protocol is to
>> > > implement
>> > > > this API, so it should give you a clear idea of what we're looking
>> for.
>> > > >
>> > > > Of course, I agree with you that the protocol should be simple and
>> > > flexible
>> > > > enough to allow other implementations for different languages and
>> > > > platforms.
>> > > >
>> > > > -Val
>> > > >
>> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky > >
>> > > > wrote:
>> > > >
>> > > > > Andrey, yep, you are right. This was just a quick idea. As for
>> me, I
>> > > just
>> > > > > don't want to repeat the same problem with compactFooter in thin
>> > client
>> > > > api
>> > > > > of ignite 2.x.
>> > > > >
>> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
>> > andrey.mashen...@gmail.com
>> > > >:
>> > > > >
>> > > > > > >
>> > > > > > > I suppose that we should describe this more verbose and
>> > explicit. I
>> > > > > > > nevertheless suggest to also consider writing values this way:
>> > > > > > > - arr of fields names (if name is missed, corresponding field
>> is
>> > > nil)
>> > > > > > > - arr of rows (row as array, length equal to fields array)
>> > > > > >
>> > > > > >
>> > > > > > Ivan,
>> > > > > > I think GET and PUT operation parameters should be consistent.
>> > > > > > With PUT operation this way may be tricky.
>> > > > > >
>> > > > > > SQL INSERT operation (which is similar PUT operation) semantic
>> > allows
>> > > > > > skipping columns that have a default value.
>> > > > > > Assume we have smth like this:
>> > > > > >
>> > > > > > CREATE TABLE t1 (
>> > > > > >'id' INT;
>> > > > > >'colname' VARCHAR DEFAULT "abc";
>>

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Pavel Tupitsyn]

Val,

> I don't think there is a significantly better way
> of doing this in Java.

Yep looks like there is no way to return two values without boxing.
No ref, no out, no value types.

> Schema already provides this information, doesn't it?

It does, though we don't have an agreement on how to expose this on public
API yet,
or do we?

On Tue, Jul 6, 2021 at 12:44 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> That's a good point, but I don't think there is a significantly better way
> of doing this in Java.
>
> There should be a way to check if a field is nullable or not though. Schema
> already provides this information, doesn't it?
>
> -Val
>
> On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > Looks like Tuple API has no efficient way to tell if a value for a
> nullable
> > column of primitive type is null.
> >
> > - Tuple#intValue() will return 0 when the actual value is null => not
> clear
> > if 0 is 0 or null.
> > - Tuple#value() works, but is more expensive due to boxing and type
> lookup.
> >
> > Any ideas on how to improve this?
> >
>

Ignite 3.0 Tuple API: how to check if value is null?

[Pavel Tupitsyn]

Igniters,

Looks like Tuple API has no efficient way to tell if a value for a nullable
column of primitive type is null.

- Tuple#intValue() will return 0 when the actual value is null => not clear
if 0 is 0 or null.
- Tuple#value() works, but is more expensive due to boxing and type lookup.

Any ideas on how to improve this?

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

I've updated the IEP to support "Live Schema" [1] from IEP-54.
Some operations now have schemaless variants, where tuples are serialized
as maps (String -> val).

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)

On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky  wrote:

> Val, my understanding about it was exactly the same as yours, but, again, I
> heard a different opinion.
>
> But nevertheless, binary protocol should not be about objects, records aka
> tuples are the best varii, simple and powerful.
>
> As for me, I want to take part in implementing python and golang thin
> clients for ignite 3, so consider my remarks using this info. I am not just
> a rude critic,
> I am just interested in consice and universal binary prorocol
> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > KV view does work over the tuples. Nested objects and arbitrary
> structures
> > can be stored as blobs. So if you need a basic KV cache, you can always
> > create a table with two blob fields - one for key and one for value - and
> > store anything there.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky 
> > wrote:
> >
> > > Val, am I right, that kv view over the tuples is just simple mapping
> from
> > > POJO to tuple? No collections, no nested objects? I have discussed this
> > in
> > > private with Igor and Pavel and they told me different info.
> > >
> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com
> > > >:
> > >
> > > > Ivan,
> > > >
> > > > I was answering your question about the KV API. The API I provided
> has
> > > been
> > > > discussed and agreed upon. One of the goals of the protocol is to
> > > implement
> > > > this API, so it should give you a clear idea of what we're looking
> for.
> > > >
> > > > Of course, I agree with you that the protocol should be simple and
> > > flexible
> > > > enough to allow other implementations for different languages and
> > > > platforms.
> > > >
> > > > -Val
> > > >
> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > Andrey, yep, you are right. This was just a quick idea. As for me,
> I
> > > just
> > > > > don't want to repeat the same problem with compactFooter in thin
> > client
> > > > api
> > > > > of ignite 2.x.
> > > > >
> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
> > andrey.mashen...@gmail.com
> > > >:
> > > > >
> > > > > > >
> > > > > > > I suppose that we should describe this more verbose and
> > explicit. I
> > > > > > > nevertheless suggest to also consider writing values this way:
> > > > > > > - arr of fields names (if name is missed, corresponding field
> is
> > > nil)
> > > > > > > - arr of rows (row as array, length equal to fields array)
> > > > > >
> > > > > >
> > > > > > Ivan,
> > > > > > I think GET and PUT operation parameters should be consistent.
> > > > > > With PUT operation this way may be tricky.
> > > > > >
> > > > > > SQL INSERT operation (which is similar PUT operation) semantic
> > allows
> > > > > > skipping columns that have a default value.
> > > > > > Assume we have smth like this:
> > > > > >
> > > > > > CREATE TABLE t1 (
> > > > > >'id' INT;
> > > > > >'colname' VARCHAR DEFAULT "abc";
> > > > > > )
> > > > > > INSERT INTO t1 VALUES(1)
> > > > > >
> > > > > > Actually, this will add a row (1, "abc")
> > > > > >
> > > > > > Your suggestion related to missed fields will not work this way
> as
> > it
> > > > is
> > > > > > impossible to distinct
> > > > > > case with 'null' value from the case with a default value.
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 1, 2021 a

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

> No it isn't, I have carefully read code and IEP, in your code you write
> schema id in each tuple.

There is no code for batch operations yet.

Here is the description of TUPLE_GET_ALL:
- UUID: table ID
- int: schema ID
- arr of arr: array of rows with values for all columns in given schema
(nil when value is missing for a column)

As you can see, schema ID is written once for all rows.
A row is just a set of values according to the schema.


> Also, my biggest concern -- extra serde step. I suppose we should pass
> bytearray to internal api, and use msgpack throughout all wire protocols,
> as tarantool does.

I agree. But this was decided before in IEP-54, and is out of scope for
current IEP.
Would you like to start a separate thread to discuss this? Or I can do this
a bit later.

On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky  wrote:

> > This is described in all operations that include multiple tuples.
> No it isn't, I have carefully read code and IEP, in your code you write
> schema id in each tuple.
>
> Also, my biggest concern -- extra serde step. I suppose we should pass
> bytearray to internal api, and use msgpack throughout all wire protocols,
> as tarantool does.
>
> чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
>
> > Ivan,
> >
> > >  that there is not neccesary to write schema versions in each row
> > > in collectionof tuples
> >
> > This is described in all operations that include multiple tuples.
> >
> >
> > > it is not clear from your code (probably
> > > mistake?) how differ key tuples and value tuples from each other
> >
> > Key tuples include only key columns. Key columns come first in the
> schema.
> > Value tuples include all columns, key and value. Added "Key tuples"
> > section.
> >
> >
> > > As for me, these excercises with schema's doesn't worth a lot
> >
> > I'll add a benchmark and we'll see.
> >
> >
> > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky 
> > wrote:
> >
> > > I suppose, that there is not neccesary to write schema versions in each
> > row
> > > in collectionof tuples. Also it is not clear from your code (probably
> > > mistake?) how differ key tuples and value tuples from each other. In
> > > readTuple you always read full schema and check for full length. As for
> > me,
> > > these excercises with schema's doesn't worth a lot. I.e. postgres just
> > > writes field names and then simpy rows with data. Saving few bytes
> > doesn't
> > > make much deal. Btw, msgpack has special types for short strings (i.e.
> > > str8). It is much easier use it and write field name as is.
> > >
> > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> > >
> > > > Ivan, tuple serialization section added to the IEP, let me know if it
> > is
> > > > clear enough.
> > > >
> > > > Thanks!
> > > >
> > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > I can't find any description of tuple serialization in IEP, only in
> > > code
> > > > >
> > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan,
> > > > > >
> > > > > > 0. The IEP is not in progress, it is ready for review and
> > discussion.
> > > > > > 1. Tuple serialization is described in the IEP and demonstrated
> in
> > > the
> > > > > PoC
> > > > > > (see ClientMessageHandler#readTuple), let me know if more details
> > are
> > > > > > required
> > > > > > 2. Tuple schema serialization is described in SCHEMAS_GET
> section.
> > > > Table
> > > > > > schema (configuration) needs more details, you are right - I'll
> add
> > > > them.
> > > > > > 3. This IEP is about tables (tuple-based) API only, since it is
> the
> > > > only
> > > > > > API that we have right now, as noted in Risks and Assumptions.
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Also, is there any clear information about KV api? Is there any
> > > plan
> > > > to
> > > > > > > implement it? Or is there any proposal about it?
> > > > > > >
> > &g

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan,

>  that there is not neccesary to write schema versions in each row
> in collectionof tuples

This is described in all operations that include multiple tuples.


> it is not clear from your code (probably
> mistake?) how differ key tuples and value tuples from each other

Key tuples include only key columns. Key columns come first in the schema.
Value tuples include all columns, key and value. Added "Key tuples" section.


> As for me, these excercises with schema's doesn't worth a lot

I'll add a benchmark and we'll see.


On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky  wrote:

> I suppose, that there is not neccesary to write schema versions in each row
> in collectionof tuples. Also it is not clear from your code (probably
> mistake?) how differ key tuples and value tuples from each other. In
> readTuple you always read full schema and check for full length. As for me,
> these excercises with schema's doesn't worth a lot. I.e. postgres just
> writes field names and then simpy rows with data. Saving few bytes doesn't
> make much deal. Btw, msgpack has special types for short strings (i.e.
> str8). It is much easier use it and write field name as is.
>
> чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
>
> > Ivan, tuple serialization section added to the IEP, let me know if it is
> > clear enough.
> >
> > Thanks!
> >
> > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky 
> > wrote:
> >
> > > I can't find any description of tuple serialization in IEP, only in
> code
> > >
> > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> > >
> > > > Ivan,
> > > >
> > > > 0. The IEP is not in progress, it is ready for review and discussion.
> > > > 1. Tuple serialization is described in the IEP and demonstrated in
> the
> > > PoC
> > > > (see ClientMessageHandler#readTuple), let me know if more details are
> > > > required
> > > > 2. Tuple schema serialization is described in SCHEMAS_GET section.
> > Table
> > > > schema (configuration) needs more details, you are right - I'll add
> > them.
> > > > 3. This IEP is about tables (tuple-based) API only, since it is the
> > only
> > > > API that we have right now, as noted in Risks and Assumptions.
> > > >
> > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > Also, is there any clear information about KV api? Is there any
> plan
> > to
> > > > > implement it? Or is there any proposal about it?
> > > > >
> > > > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> > > > >
> > > > > > Pavel, but IEP is in progress, isn't it?
> > > > > >
> > > > > > 1. There is not any information about tuple serialization. And
> > there
> > > > > isn't
> > > > > > a clear consensus about it.
> > > > > > 2. There is not any information about schrma serialization
> format.
> > > And
> > > > > > AFAIK, there isn't a clear consensus also.
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > > > >
> > > > > >> Igniters,
> > > > > >>
> > > > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > > > >> PoC is in progress [2]
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > > > >>
> > > > > >> [2] https://github.com/apache/ignite-3/pull/191
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan, tuple serialization section added to the IEP, let me know if it is
clear enough.

Thanks!

On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky  wrote:

> I can't find any description of tuple serialization in IEP, only in code
>
> чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
>
> > Ivan,
> >
> > 0. The IEP is not in progress, it is ready for review and discussion.
> > 1. Tuple serialization is described in the IEP and demonstrated in the
> PoC
> > (see ClientMessageHandler#readTuple), let me know if more details are
> > required
> > 2. Tuple schema serialization is described in SCHEMAS_GET section. Table
> > schema (configuration) needs more details, you are right - I'll add them.
> > 3. This IEP is about tables (tuple-based) API only, since it is the only
> > API that we have right now, as noted in Risks and Assumptions.
> >
> > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky 
> > wrote:
> >
> > > Also, is there any clear information about KV api? Is there any plan to
> > > implement it? Or is there any proposal about it?
> > >
> > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> > >
> > > > Pavel, but IEP is in progress, isn't it?
> > > >
> > > > 1. There is not any information about tuple serialization. And there
> > > isn't
> > > > a clear consensus about it.
> > > > 2. There is not any information about schrma serialization format.
> And
> > > > AFAIK, there isn't a clear consensus also.
> > > >
> > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > >
> > > >> Igniters,
> > > >>
> > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > >> PoC is in progress [2]
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > >>
> > > >> [2] https://github.com/apache/ignite-3/pull/191
> > > >>
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan,

0. The IEP is not in progress, it is ready for review and discussion.
1. Tuple serialization is described in the IEP and demonstrated in the PoC
(see ClientMessageHandler#readTuple), let me know if more details are
required
2. Tuple schema serialization is described in SCHEMAS_GET section. Table
schema (configuration) needs more details, you are right - I'll add them.
3. This IEP is about tables (tuple-based) API only, since it is the only
API that we have right now, as noted in Risks and Assumptions.

On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky  wrote:

> Also, is there any clear information about KV api? Is there any plan to
> implement it? Or is there any proposal about it?
>
> чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
>
> > Pavel, but IEP is in progress, isn't it?
> >
> > 1. There is not any information about tuple serialization. And there
> isn't
> > a clear consensus about it.
> > 2. There is not any information about schrma serialization format. And
> > AFAIK, there isn't a clear consensus also.
> >
> > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> >
> >> Igniters,
> >>
> >> Please review the IEP for thin client protocol in 3.0 [1].
> >> PoC is in progress [2]
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> >>
> >> [2] https://github.com/apache/ignite-3/pull/191
> >>
> >
>

IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

Please review the IEP for thin client protocol in 3.0 [1].
PoC is in progress [2]

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0

[2] https://github.com/apache/ignite-3/pull/191

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Pavel Tupitsyn]

Val, agreed.
Let's add length(), value(index), and Iterable to the Tuple interface.

I've updated the ticket: https://issues.apache.org/jira/browse/IGNITE-14342

On Wed, Jun 30, 2021 at 11:17 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> Thanks for your response, makes sense.
>
> Regarding the values() method: I would instead add the required methods to
> the Tuple itself. E.g., it can implement Iterable, and additionally have
> the value(int index) method, plus anything else that we might need.
> I don't like returning a collection, because in Java it's a much wider API.
> We will end up returning an implementation that throws
> UnsupportedOperationException for most of the methods. I know this approach
> is not uncommon in the Java world, but this doesn't make it good.
> In .NET and other languages, we can use other abstractions, of course.
> Every platform has its own specifics and practices, so APIs don't have to
> be identical.
>
> -Val
>
> On Wed, Jun 30, 2021 at 7:44 AM Pavel Tupitsyn 
> wrote:
>
> > Hi Andrey,
> >
> > > This will force us to bother about interfaces/contracts and
> compatibility
> > > aspects in the future
> >
> > Schemas and versions are part of thin client wire protocol.
> > This protocol is a public API - we'll have to care about compatibility
> > anyway.
> >
> > Schema evolution is an important feature that users should understand.
> > I see no reason to hide it from the API.
> >
> >
> > > We already have a ticket [1]...
> > > Will 'idx -> column' mapping only be enough for your purposes?
> >
> > The ticket sounds reasonable, but there are no API details.
> > At the very least we need public access to column names, types, and
> > indexes.
> > I propose something like this:
> >
> > interface Tuple {
> > TupleSchema getSchema();
> > }
> >
> > class TupleSchema {
> > int getVersion();
> > List getColumns();
> > }
> >
> > class TupleSchemaColumn {
> > int index;
> > String name;
> > String typeName;
> > bool isKey;
> > }
> >
> > On Wed, Jun 30, 2021 at 11:05 AM Andrey Mashenkov <
> > andrey.mashen...@gmail.com> wrote:
> >
> > > Hi Pavel,
> > >
> > > 2. Schema and its version are internal things and shouldn't be exposed,
> > > otherwise, eventually, it will lead to the user will manage schemas
> > > manually on his side for some purposes.
> > > This will force us to bother about interfaces/contracts and
> compatibility
> > > aspects in the future with uncertain benefits for a user.
> > >
> > > As SchemaDescriptor is an internal API and the user expects a public
> API.
> > > I don't think we want to convert the descriptor back into public API
> > terms
> > > and it may be impossible for some data.
> > >
> > > We already have a ticket [1] to support accessing a column by an index
> > and
> > > exposing some metadata.
> > > Will 'idx -> column' mapping only be enough for your purposes?
> > >
> > > > int Tuple.columnIndex(String)
> > >
> > >
> > >
> > >  [1] https://issues.apache.org/jira/browse/IGNITE-14342
> > >
> > > On Wed, Jun 30, 2021 at 9:34 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Hi Val,
> > > >
> > > > Thanks for the comments, please see below:
> > > >
> > > >
> > > > > Users will identify tables using names, they will never use IDs
> > > >
> > > > Ok, let's keep it this way.
> > > >
> > > >
> > > > > Sounds like the Tuple should implement Iterable.
> > > >
> > > > I don't think Iterable is enough.
> > > > We should have a way to get values by column index: Tuple.value(int
> > > index),
> > > > where index is according to column order in schema.
> > > >
> > > > Combined with Tuple.schema(), it will provide an efficient way to
> > consume
> > > > data -
> > > > users will be able to read columns in any order, skip some of them,
> > etc.
> > > >
> > > > This is a common pattern in APIs like ODBC or System.Data [1],
> > > > which we'll need to implement on top of our thin clients later.
> > > >
> > > >
> > > > > However, whether 

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Pavel Tupitsyn]

Hi Andrey,

> This will force us to bother about interfaces/contracts and compatibility
> aspects in the future

Schemas and versions are part of thin client wire protocol.
This protocol is a public API - we'll have to care about compatibility
anyway.

Schema evolution is an important feature that users should understand.
I see no reason to hide it from the API.


> We already have a ticket [1]...
> Will 'idx -> column' mapping only be enough for your purposes?

The ticket sounds reasonable, but there are no API details.
At the very least we need public access to column names, types, and indexes.
I propose something like this:

interface Tuple {
TupleSchema getSchema();
}

class TupleSchema {
int getVersion();
List getColumns();
}

class TupleSchemaColumn {
int index;
String name;
String typeName;
bool isKey;
}

On Wed, Jun 30, 2021 at 11:05 AM Andrey Mashenkov <
andrey.mashen...@gmail.com> wrote:

> Hi Pavel,
>
> 2. Schema and its version are internal things and shouldn't be exposed,
> otherwise, eventually, it will lead to the user will manage schemas
> manually on his side for some purposes.
> This will force us to bother about interfaces/contracts and compatibility
> aspects in the future with uncertain benefits for a user.
>
> As SchemaDescriptor is an internal API and the user expects a public API.
> I don't think we want to convert the descriptor back into public API terms
> and it may be impossible for some data.
>
> We already have a ticket [1] to support accessing a column by an index and
> exposing some metadata.
> Will 'idx -> column' mapping only be enough for your purposes?
>
> > int Tuple.columnIndex(String)
>
>
>
>  [1] https://issues.apache.org/jira/browse/IGNITE-14342
>
> On Wed, Jun 30, 2021 at 9:34 AM Pavel Tupitsyn 
> wrote:
>
> > Hi Val,
> >
> > Thanks for the comments, please see below:
> >
> >
> > > Users will identify tables using names, they will never use IDs
> >
> > Ok, let's keep it this way.
> >
> >
> > > Sounds like the Tuple should implement Iterable.
> >
> > I don't think Iterable is enough.
> > We should have a way to get values by column index: Tuple.value(int
> index),
> > where index is according to column order in schema.
> >
> > Combined with Tuple.schema(), it will provide an efficient way to consume
> > data -
> > users will be able to read columns in any order, skip some of them, etc.
> >
> > This is a common pattern in APIs like ODBC or System.Data [1],
> > which we'll need to implement on top of our thin clients later.
> >
> >
> > > However, whether a user might
> > > need to get a table schema for a particular version, I'm not sure. Do
> you
> > > have a use case in mind for this? If not, I would keep this internal
> >
> > Ok, we can keep the Table.schema(ver) method internal, as long as
> > Table.schemas() is public and includes schema versions.
> >
> >
> > > We already have async counterparts for all the methods where this is
> > applicable
> >
> > IgniteTables.createTable(), dropTable(), tables(), table() do not have
> > async counterparts,
> > while internally they perform blocking wait on some futures.
> >
> >
> > [1]
> >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.data.idatareader?view=net-5.0
> >
> > On Wed, Jun 30, 2021 at 4:51 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Hi Pavel,
> > >
> > > Please see my comments below.
> > >
> > > -Val
> > >
> > > On Tue, Jun 29, 2021 at 2:23 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > While working on "IEP-76 Thin Client Protocol for Ignite 3.0" [1] (to
> > be
> > > > discussed separately), the following suggestions for the Table API
> came
> > > up:
> > > >
> > > > 1. Expose table IDs: sending table name with every operation (e.g.
> GET)
> > > is
> > > > inefficient, string serialization is expensive by itself and names
> can
> > be
> > > > long.
> > > > - Table.id()
> > > > - IgniteTables.table(UUID)
> > > > - IgniteTables.dropTable(UUID)
> > > >
> > >
> > > I don't think this should be a part of the public API. Users will
> > identify
> > > tables using names, they will never use IDs. As an internal
> optimization
> > >

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Pavel Tupitsyn]

Hi Val,

Thanks for the comments, please see below:


> Users will identify tables using names, they will never use IDs

Ok, let's keep it this way.


> Sounds like the Tuple should implement Iterable.

I don't think Iterable is enough.
We should have a way to get values by column index: Tuple.value(int index),
where index is according to column order in schema.

Combined with Tuple.schema(), it will provide an efficient way to consume
data -
users will be able to read columns in any order, skip some of them, etc.

This is a common pattern in APIs like ODBC or System.Data [1],
which we'll need to implement on top of our thin clients later.


> However, whether a user might
> need to get a table schema for a particular version, I'm not sure. Do you
> have a use case in mind for this? If not, I would keep this internal

Ok, we can keep the Table.schema(ver) method internal, as long as
Table.schemas() is public and includes schema versions.


> We already have async counterparts for all the methods where this is
applicable

IgniteTables.createTable(), dropTable(), tables(), table() do not have
async counterparts,
while internally they perform blocking wait on some futures.


[1]
https://docs.microsoft.com/en-us/dotnet/api/system.data.idatareader?view=net-5.0

On Wed, Jun 30, 2021 at 4:51 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Pavel,
>
> Please see my comments below.
>
> -Val
>
> On Tue, Jun 29, 2021 at 2:23 PM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > While working on "IEP-76 Thin Client Protocol for Ignite 3.0" [1] (to be
> > discussed separately), the following suggestions for the Table API came
> up:
> >
> > 1. Expose table IDs: sending table name with every operation (e.g. GET)
> is
> > inefficient, string serialization is expensive by itself and names can be
> > long.
> > - Table.id()
> > - IgniteTables.table(UUID)
> > - IgniteTables.dropTable(UUID)
> >
>
> I don't think this should be a part of the public API. Users will identify
> tables using names, they will never use IDs. As an internal optimization
> though - sure, we can have that. Sounds similar to the cache ID in 2.x.
>
>
> >
> > 2. Expose tuple schemas: to reduce payload size when sending tuples to
> the
> > client, we'll write only the schema version and column values, then the
> > client can retrieve and cache schemas (ordered set of columns per
> version).
> > - Tuple.schema()
> > - Table.schemas()
> > - Table.schema(ver)
> >
>
> Exposing the schema of a tuple makes sense. However, whether a user might
> need to get a table schema for a particular version, I'm not sure. Do you
> have a use case in mind for this? If not, I would keep this internal as
> well.
>
>
> >
> > 3. Expose tuple values as a collection: to serialize tuples efficiently
> > (see p2) we need an API to get all values at once. Right now the only API
> > is to get values by column name, which involves a HashMap lookup on every
> > call.
> > - Tuple.values()
> >
>
> Sounds like the Tuple should implement Iterable.
>
>
> >
> > 4. Simplify createTable API: use POJO-based configuration.
> > Creating a Consumer when some properties are optional seems to be
> > non-trivial.
> >
>
> Yes, it's currently convoluted for sure. To my knowledge, there are plans
> to improve this, but I will let other folks chime in with more details.
>
>
> >
> > 5. Add async methods to IgniteTables API (all methods are async inside
> > anyway)
> >
>
> We already have async counterparts for all the methods where this is
> applicable. E.g., for TableView#get, there is TableView#getAsync. Is there
> anything else that you propose to add?
>
>
> >
> >
> > Thoughts, objections?
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> >
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> >
>

Ignite 3.0 IgniteTables API Improvement Suggestion

[Pavel Tupitsyn]

Igniters,

While working on "IEP-76 Thin Client Protocol for Ignite 3.0" [1] (to be
discussed separately), the following suggestions for the Table API came up:

1. Expose table IDs: sending table name with every operation (e.g. GET) is
inefficient, string serialization is expensive by itself and names can be
long.
- Table.id()
- IgniteTables.table(UUID)
- IgniteTables.dropTable(UUID)

2. Expose tuple schemas: to reduce payload size when sending tuples to the
client, we'll write only the schema version and column values, then the
client can retrieve and cache schemas (ordered set of columns per version).
- Tuple.schema()
- Table.schemas()
- Table.schema(ver)

3. Expose tuple values as a collection: to serialize tuples efficiently
(see p2) we need an API to get all values at once. Right now the only API
is to get values by column name, which involves a HashMap lookup on every
call.
- Tuple.values()

4. Simplify createTable API: use POJO-based configuration.
Creating a Consumer when some properties are optional seems to be
non-trivial.

5. Add async methods to IgniteTables API (all methods are async inside
anyway)


Thoughts, objections?


[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0

[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach

Re: Contributor Permission

[Pavel Tupitsyn]

Hi Saurabh,

I've added you to the Contributors role in JIRA.
Welcome to the Apache Ignite community!

On Tue, Jun 29, 2021 at 10:34 AM saurabh chhajed 
wrote:

> Hi Team,
>
> I wanted to contribute to some ML based inference code for Ignite, and
> Requesting to be added as contributor permission in Apache Ignite JIRA, if
> possible. Thanks for your consideration, here are my details -
>
> Username: schhajed
> Email: saurabh.chha...@gmail.com
> Full Name: Saurabh Chhajed
>
>
> Regards,
>

Re: [VOTE] Release Apache Ignite 3.0.0-alpha2 RC1

[Pavel Tupitsyn]

+1 (binding)

On Mon, Jun 28, 2021 at 8:23 PM Вячеслав Коптилин 
wrote:

> +1
>
> Thanks,
> S.
>
> пн, 28 июн. 2021 г. в 18:09, Igor Sapego :
>
> > +1
> >
> > Best Regards,
> > Igor
> >
> >
> > On Sat, Jun 26, 2021 at 1:41 AM Nikita Ivanov 
> wrote:
> >
> > > +1
> > >
> > > --
> > > Nikita Ivanov
> > >
> > >
> > >
> > > On Fri, Jun 25, 2021 at 3:31 PM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Dear Community,
> > > >
> > > > In the last several months, the development of Ignite 3 has been
> moving
> > > > forward significantly. On top of what we already had in the first
> > Alpha,
> > > we
> > > > have the following features ready:
> > > >
> > > > - Replication infrastructure based on Raft
> > > > - In-memory atomic storage with the basic insert-read functionality
> > > > - New schema management engine and API
> > > >
> > > > In my view, this constitutes a significant milestone, and I propose
> to
> > > > release it as the Alpha 2. This way it will be available for
> download,
> > > and
> > > > anyone will be able to play with the project, run examples, get a
> feel
> > of
> > > > how Ignite will work in the future, and provide feedback.
> > > >
> > > > Please vote.
> > > >
> > > > ---
> > > >
> > > > The release candidate is uploaded here:
> > > > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha2-rc1/
> > > > Maven staging:
> > > >
> > https://repository.apache.org/content/repositories/orgapacheignite-1522/
> > > > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha2-rc1
> > > >
> > > > Jira tickets:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%2012315922%20AND%20fixVersion%20%3D%2012349574%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC
> > > >
> > > > DEVNOTES:
> > > > https://github.com/apache/ignite-3/blob/3.0.0-alpha2-rc1/DEVNOTES.md
> > > >
> > > > The vote is formal, see voting guidelines:
> > > > https://www.apache.org/foundation/voting.html
> > > >
> > > > +1 - accept Apache Ignite 3.0.0-alpha2 RC1
> > > > 0 - don't care either way
> > > > -1 - DO NOT accept Apache Ignite 3.0.0-alpha2 RC1 (explain why)
> > > >
> > > > See notes on how to verify release here:
> > > > https://www.apache.org/info/verification.html
> > > >
> > > > This vote will be open for 72 hours and will close on June 28th at
> 11pm
> > > > UTC:
> > > >
> > > >
> > >
> >
> https://www.timeanddate.com/countdown/to?iso=20210628T16&p0=224&msg=Apache+Ignite+3.0.0-alpha2+RC1&font=cursive&csz=1
> > > >
> > > > -Val
> > > >
> > >
> >
>

Re: IEP-75 Thin Client MsgPack Serialization for 3.0

[Pavel Tupitsyn]

Ivan,

Yes, this proposal is about the binary protocol - the way we serialize
primitive values,
the foundation for the thin client protocol.

Key-value API still has some uncertainty around it, we'll discuss it
separately.


On Wed, Jun 23, 2021 at 2:31 PM Ivan Daschinsky  wrote:

> Pavel, let me clarify one thing.
>
> 1. If this proposal is about binary protocol, then there is no objection I
> suppose.
> 2. If this proposal about serialization of key-value, there are some
> uncertainties, especially about complex objects. In this case this proposal
> needs more work.
>
> ср, 23 июн. 2021 г. в 14:07, Pavel Tupitsyn :
>
> > Igniters,
> >
> > Looks like there are no objections and we can accept the proposal.
> > I will close it tomorrow and move on to the thin client protocol itself.
> >
> > On Fri, Jun 18, 2021 at 12:10 PM Ivan Daschinsky 
> > wrote:
> >
> > > >> To make it fair. Ignite uses thread-local reusable buffers, see [1].
> > > I know, but PooledMessageBufferOutput is not about thread-local, isn't
> > it?
> > >
> > > I'm not against about MsgPack, I'm for fair and not biased comparison.
> > >
> > > I suppose that MsgPack is an ideal candidate for thin client binary
> > > protocol, not only for serializing some user data.
> > > I am analyzed some tarantool connectors [1] [2] [3] and found msgpack
> > based
> > > protocol is a really good idea.
> > > Also there is realy super fast and just 1 header library with liberal
> > BSD-2
> > > licence for C -- msgpuck [4]. It used in tarantool itself
> > > and in [1] and is stable and bullet proof.
> > >
> > > [1] -- https://github.com/igorcoding/asynctnt
> > > [2] -- https://github.com/tarantool/tarantool-python/
> > > [3] -- https://github.com/tarantool/go-tarantool
> > > [4] -- https://github.com/rtsisyk/msgpuck
> > >
> > > пт, 18 июн. 2021 г. в 11:44, Pavel Tupitsyn :
> > >
> > > > Ivan,
> > > >
> > > > > why do you use  PooledMessageBufferOutput in benchmarks?
> > > >
> > > > To make it fair. Ignite uses thread-local reusable buffers, see [1].
> > > >
> > > >
> > > > > why packer from msgpack-core show better performance than
> > > > > BinaryWriter. And I suppose that benchmark is not quite fair.
> > > >
> > > > MsgPack writes and reads less bytes, so it should be faster.
> > > > Benchmark is not 100% fair, there are some small extra things that
> > > > BinaryWriter does.
> > > >
> > > > However:
> > > > 1. I don't think we care about super-precise benchmarks here, just
> the
> > > > ballpark.
> > > > 2. We are discussing the format, not the implementation.
> > > >
> > > > Important takeaway is:
> > > > The format does not prevent someone from implementing it efficiently.
> > > >
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryWriterExImpl.java#L101
> > > >
> > > > On Thu, Jun 17, 2021 at 10:40 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > wrote:
> > > >
> > > > > Andrey, here we discuss serialization format, as far as I
> understand.
> > > > > Current implementation of ignite binary object serialization can be
> > > > > rewritten.
> > > > > If we do not care about fast (O(1)) field lookup, about schema
> > > validation
> > > > > and so on, msgpack is a really good option. It is also good for
> > client
> > > > > binary protocol, i.e.
> > > > > tarantool uses it.
> > > > >
> > > > > >> Binarilizable interface forces user to write serialization code
> > > > > I am talking about speed comparison. You can see from Pavel's data,
> > > > > jackson-msgpack shows a pathetic performance comparing with a
> > ignite's
> > > > > default binary marshaller. If you want really fast serialization --
> > the
> > > > > only option is to write code by yourself or use code generation.
> > > Default
> > > > > packer from msgpack-core java package is similar to BinaryWriter.
> So
> > I
> > > am
> > > > > wondering why packer from msgpack-core 

Re: IEP-75 Thin Client MsgPack Serialization for 3.0

[Pavel Tupitsyn]

Igniters,

Looks like there are no objections and we can accept the proposal.
I will close it tomorrow and move on to the thin client protocol itself.

On Fri, Jun 18, 2021 at 12:10 PM Ivan Daschinsky 
wrote:

> >> To make it fair. Ignite uses thread-local reusable buffers, see [1].
> I know, but PooledMessageBufferOutput is not about thread-local, isn't it?
>
> I'm not against about MsgPack, I'm for fair and not biased comparison.
>
> I suppose that MsgPack is an ideal candidate for thin client binary
> protocol, not only for serializing some user data.
> I am analyzed some tarantool connectors [1] [2] [3] and found msgpack based
> protocol is a really good idea.
> Also there is realy super fast and just 1 header library with liberal BSD-2
> licence for C -- msgpuck [4]. It used in tarantool itself
> and in [1] and is stable and bullet proof.
>
> [1] -- https://github.com/igorcoding/asynctnt
> [2] -- https://github.com/tarantool/tarantool-python/
> [3] -- https://github.com/tarantool/go-tarantool
> [4] -- https://github.com/rtsisyk/msgpuck
>
> пт, 18 июн. 2021 г. в 11:44, Pavel Tupitsyn :
>
> > Ivan,
> >
> > > why do you use  PooledMessageBufferOutput in benchmarks?
> >
> > To make it fair. Ignite uses thread-local reusable buffers, see [1].
> >
> >
> > > why packer from msgpack-core show better performance than
> > > BinaryWriter. And I suppose that benchmark is not quite fair.
> >
> > MsgPack writes and reads less bytes, so it should be faster.
> > Benchmark is not 100% fair, there are some small extra things that
> > BinaryWriter does.
> >
> > However:
> > 1. I don't think we care about super-precise benchmarks here, just the
> > ballpark.
> > 2. We are discussing the format, not the implementation.
> >
> > Important takeaway is:
> > The format does not prevent someone from implementing it efficiently.
> >
> >
> >
> > [1]
> >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryWriterExImpl.java#L101
> >
> > On Thu, Jun 17, 2021 at 10:40 PM Ivan Daschinsky 
> > wrote:
> >
> > > Andrey, here we discuss serialization format, as far as I understand.
> > > Current implementation of ignite binary object serialization can be
> > > rewritten.
> > > If we do not care about fast (O(1)) field lookup, about schema
> validation
> > > and so on, msgpack is a really good option. It is also good for client
> > > binary protocol, i.e.
> > > tarantool uses it.
> > >
> > > >> Binarilizable interface forces user to write serialization code
> > > I am talking about speed comparison. You can see from Pavel's data,
> > > jackson-msgpack shows a pathetic performance comparing with a ignite's
> > > default binary marshaller. If you want really fast serialization -- the
> > > only option is to write code by yourself or use code generation.
> Default
> > > packer from msgpack-core java package is similar to BinaryWriter. So I
> am
> > > wondering why packer from msgpack-core show better performance than
> > > BinaryWriter. And I suppose that benchmark is not quite fair.
> > >
> > >
> > > чт, 17 июн. 2021 г. в 22:19, Andrey Mashenkov <
> > andrey.mashen...@gmail.com
> > > >:
> > >
> > > > Ivan, thankd for clarification.
> > > >
> > > > Binarilizable interface forces user to write serialization code. We
> can
> > > > support this or similar interface.
> > > > But I'd like Ignite has some default serializer in addition. It can
> be
> > > also
> > > > useful e.g. in compute for param and result serialization.
> > > >
> > > > BinaryObjectBuider requires an Ignite node for object construction,
> but
> > > we
> > > > are looking for a detached builder and won't care about schemas.
> > > >
> > > > AFAIR, BinaryObject creates an objectReader on every single field
> read
> > > > operation.
> > > > So, BO solution produces a lot of garbage and BO has noticable
> overhead
> > > > which affects the object footprint.
> > > >
> > > > чт, 17 июн. 2021 г., 21:41 Ivan Daschinsky :
> > > >
> > > > > >> Double checked -- there is not any links to PR either in IEP or
> in
> > > > jira
> > > > > issue
> > > > > Sorry, there is a link in IEP, but not in jira ticket.
> &

Re: IEP-75 Thin Client MsgPack Serialization for 3.0

[Pavel Tupitsyn]

Ivan,

> why do you use  PooledMessageBufferOutput in benchmarks?

To make it fair. Ignite uses thread-local reusable buffers, see [1].


> why packer from msgpack-core show better performance than
> BinaryWriter. And I suppose that benchmark is not quite fair.

MsgPack writes and reads less bytes, so it should be faster.
Benchmark is not 100% fair, there are some small extra things that
BinaryWriter does.

However:
1. I don't think we care about super-precise benchmarks here, just the
ballpark.
2. We are discussing the format, not the implementation.

Important takeaway is:
The format does not prevent someone from implementing it efficiently.



[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryWriterExImpl.java#L101

On Thu, Jun 17, 2021 at 10:40 PM Ivan Daschinsky 
wrote:

> Andrey, here we discuss serialization format, as far as I understand.
> Current implementation of ignite binary object serialization can be
> rewritten.
> If we do not care about fast (O(1)) field lookup, about schema validation
> and so on, msgpack is a really good option. It is also good for client
> binary protocol, i.e.
> tarantool uses it.
>
> >> Binarilizable interface forces user to write serialization code
> I am talking about speed comparison. You can see from Pavel's data,
> jackson-msgpack shows a pathetic performance comparing with a ignite's
> default binary marshaller. If you want really fast serialization -- the
> only option is to write code by yourself or use code generation. Default
> packer from msgpack-core java package is similar to BinaryWriter. So I am
> wondering why packer from msgpack-core show better performance than
> BinaryWriter. And I suppose that benchmark is not quite fair.
>
>
> чт, 17 июн. 2021 г. в 22:19, Andrey Mashenkov  >:
>
> > Ivan, thankd for clarification.
> >
> > Binarilizable interface forces user to write serialization code. We can
> > support this or similar interface.
> > But I'd like Ignite has some default serializer in addition. It can be
> also
> > useful e.g. in compute for param and result serialization.
> >
> > BinaryObjectBuider requires an Ignite node for object construction, but
> we
> > are looking for a detached builder and won't care about schemas.
> >
> > AFAIR, BinaryObject creates an objectReader on every single field read
> > operation.
> > So, BO solution produces a lot of garbage and BO has noticable overhead
> > which affects the object footprint.
> >
> > чт, 17 июн. 2021 г., 21:41 Ivan Daschinsky :
> >
> > > >> Double checked -- there is not any links to PR either in IEP or in
> > jira
> > > issue
> > > Sorry, there is a link in IEP, but not in jira ticket.
> > >
> > > чт, 17 июн. 2021 г. в 21:39, Ivan Daschinsky :
> > >
> > > > Andrey,
> > > > >> arbitrary object graph
> > > > Also, that is not true, msgpack format doesn't handle circular
> graphs.
> > > > Think about msgpack as binary json. You couldn't understand full
> > > structure
> > > > of message if you didn't deserialize it fully before, maps and arrays
> > are
> > > > serialized just as contiguos chunks
> > > >  of values/kv-pairs. Msgpack is a really dumb and simple format.
> > > >
> > > > Also, as for me, I cannot understand why current ignite serialization
> > > > (BinaryObjectBuilder or Binarilizable) is slower than raw message
> pack
> > > > serializer.
> > > > I suppose that this is an issue and we should investigate it.
> > > >
> > > > Pavel,  why do you use  PooledMessageBufferOutput in benchmarks? I'm
> > > > sorry, but is it fair to use it?
> > > >
> > > > >> The code is linked in the IEP [2]
> > > > Double checked -- there is not any links to PR either in IEP or in
> jira
> > > > issue
> > > >
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: IEP-75 Thin Client MsgPack Serialization for 3.0

[Pavel Tupitsyn]

Ivan,

> Have you considered format with schema?

1. We should be able to serialize arbitrary user data on the client side.
I think we don't want to require extra steps from the user.

2. MsgPack can be also used in a schemaful way, when user objects are
written as arrays, not as maps - so that field names are not included.


> we should benchmark formats thoroughly

Strictly speaking, the IEP is about the format (the spec), not about the
implementation.
The format itself is simple and efficient, there is nothing to make it
slower than anything else.
C# impl proves this by beating every competitor [1].


> Could you please share your code for benchmarks?

The code is linked in the IEP [2]


[1]
https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/
[2] https://github.com/apache/ignite/pull/9178

On Thu, Jun 17, 2021 at 4:02 PM Ivan Daschinsky  wrote:

> Could you please share your code for benchmarks?
>
> чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky :
>
> > Hi, Pavel. Have you considered format with schema? Or schemaless of a
> > candidate format was  a prerequisite?
> > As for me, msgpack is great, but I suppose that we should benchmark
> > formats thoroughly. And not only for Java.
> >
> > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn :
> >
> >> Igniters,
> >>
> >> I have drafted an IEP on thin client serialization format [1],
> >> please review and let me know what you think.
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization
> >>
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

IEP-75 Thin Client MsgPack Serialization for 3.0

[Pavel Tupitsyn]

Igniters,

I have drafted an IEP on thin client serialization format [1],
please review and let me know what you think.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization

Re: [VOTE] Release pyignite 0.5.0-rc1

[Pavel Tupitsyn]

+1

Checked pip install from tar.gz on Python 3.8 on Ubuntu 20.04, ran some of
the examples.

On Thu, Jun 17, 2021 at 2:32 PM Igor Sapego  wrote:

> +1 from me
>
> Best Regards,
> Igor
>
>
> On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky 
> wrote:
>
> > +1 From me
> > Checked on Ubuntu 20.04 and windows 10
> > 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
> > 2. Native module work
> > 3. Examples
> >
> > Checked on Ubuntu 20.04 building from source package and correct work of
> > result package.
> >
> > Checked all sha256 checksums and gpg signatures.
> >
> > Let's extend voting period till June 18, 15:00 UTC
> >
> >
> >
> > ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky :
> >
> > > The vote will end at June, 17 15:00 UTC.
> > >
> > > ср, 16 июн. 2021 г. в 17:33, Ivan Daschinsky :
> > > >
> > > > Dear Igniters!
> > > >
> > > > Release candidate binaries for subj are uploaded and ready for vote
> > > > You can find them here:
> > > > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.0-rc1
> > > >
> > > > If you follow the link above, you will find source package (*.tar.gz
> > and
> > > *.zip)
> > > > and binary packages (wheels) for windows (amd64) and linux (x86_64)
> > > > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg
> signatures.
> > > > Code signing keys can be found here --
> > > https://downloads.apache.org/ignite/KEYS
> > > > Here you can find instructions how to verify packages
> > > > https://www.apache.org/info/verification.html
> > > >
> > > > You can install binary package for specific version of python using
> pip
> > > > For example do this on linux for python 3.8
> > > > >> pip install pyignite-0.5.0-cp38-cp38-manylinux1_x86_64.whl
> > > >
> > > > You can build and install package from source using this command:
> > > > >> pip install pyignite-0.5.0.tar.gz
> > > > You can build wheel on your platform using this command:
> > > > >> pip wheel --no-deps pyignite-0.5.0.tar.gz
> > > >
> > > > For building C module, you should have python headers and C compiler
> > > installed.
> > > > (i.e. for ubuntu sudo apt install build-essential python3-dev)
> > > > In Mac OS X xcode-tools and python from homebrew are the best option.
> > > >
> > > > In order to check whether C module works, use following:
> > > > >> from pyignite import _cutils
> > > > >> print(_cutils.hashcode('test'))
> > > > >> 3556498
> > > >
> > > > You can find documentation here:
> > > >
> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1
> > > >
> > > > You can find examples here (to check them, you should start ignite
> > > locally):
> > > >
> > >
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1/examples.html
> > > > Also, examples can be found in source archive in examples subfolder.
> > > > docker-compose.yml is supplied in order to start ignite quickly. (Use
> > > > `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> > > > down` to shut down it)
> > > >
> > > > Release notes:
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9d2ae81af2de22ce9e8c9d3b7ece14dd9e75ca0e;hb=61c83cb0ab6752f019518b4a2cb0724bd027755f
> > > >
> > > > Git release tag was created:
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.0.rc1
> > > >
> > > > The vote is formal, see voting guidelines
> > > > https://www.apache.org/foundation/voting.html
> > > >
> > > > +1 - to accept pyignite-0.5.0-rc1
> > > > 0 - don't care either way
> > > > -1 - DO NOT accept pyignite-0.5.0-rc1
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>

Re: New committer: Vladimir Ermakov

[Pavel Tupitsyn]

Hi Vladimir,

Welcome to the Apache Ignite community!
I've added you to the Contributors role in JIRA.

Note: "Committer" is a community member with direct commit access [1].

[1] https://apache.org/foundation/glossary.html#Committer

On Mon, Jun 7, 2021 at 4:27 PM Владимир Ермаков 
wrote:

> Hello Ignite Community!
>
> My name is Vladimir, I am a software engineer from Saint-Petersburg,
> Russia. I want to contribute to Apache Ignite.
> I am going to start with this issue - IGNITE-14120. My JIRA username is
> vermakov. Please, provide me with access to  Apache Ignite JIRA.
>
> Thanks!
>

Re: [DISCUSSION] Code style. Variable abbrevations

[Pavel Tupitsyn]

In my opinion, we should remove this rule.
Looks like a waste of time. freq or frequency, cnt or count, it is fine
either way.

On Fri, Jun 4, 2021 at 7:39 PM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> Right now, we have the rule to use some predefined list of abbrevation for
> variable names [1].
> Some of the reviewers ask to follow this rule strictly.
>
> > It is required to use abbreviated form for code consistency.
>
> I tried to implement this rule in form of checkstyle check [2] and it
> seems like many of use doesn’t follow this requirement.
> My check found 4124 violation in core module.
>
> Should we make this rule optional in documentation or should we remove it
> completely?
> Or should someone proceed and fix all the violations?
>
> WDYT?
>
>
> Example of output:
> ```
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:94:
> Abbrevation should be used for CACHE_NAME_LOCAL_ATOMIC! Please, use loc,
> instead of LOCAL [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:97:
> Abbrevation should be used for CACHE_NAME_LOCAL_TX! Please, use loc,
> instead of LOCAL [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:264:
> Abbrevation should be used for checkpointManager! Please, use mgr, instead
> of Manager [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java:63:
> Abbrevation should be used for DEFAULT_REGION! Please, use dflt, instead of
> DEFAULT [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWholeClusterRestartTest.java:47:
> Abbrevation should be used for ENTRIES_COUNT! Please, use cnt, instead of
> COUNT [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsRebalancingOnNotStableTopologyTest.java:49:
> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq,
> instead of FREQUENCY [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:75:
> Abbrevation should be used for MAX_KEY_COUNT! Please, use cnt, instead of
> COUNT [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:78:
> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq,
> instead of FREQUENCY [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/SlowHistoricalRebalanceSmallHistoryTest.java:57:
> Abbrevation should be used for SUPPLY_MESSAGE_LATCH! Please, use msg,
> instead of MESSAGE [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgniteLogicalRecoveryTest.java:74:
> Abbrevation should be used for SHARED_GROUP_NAME! Please, use grp, instead
> of GROUP [IgniteAbbrevationsRule]
> [ERROR]
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgniteLogicalRecoveryTest.java:200:
> Abbrevation should be used for cacheLoader! Please, use ldr, instead of
> Loader [IgniteAbbrevationsRule]
> ```
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules#AbbreviationRules-VariableAbbreviation
> [2] https://github.com/apache/ignite/pull/9153
>
>

Re: Unable to edit fixVersion in jira tickets

[Pavel Tupitsyn]

It is in Admin panel:
https://issues.apache.org/jira/plugins/servlet/project-config/IGNITE/administer-versions?status=unreleased

Not sure if Contributors can add new versions, so I've added 2.12 there.

On Fri, Jun 4, 2021 at 2:41 PM Alexey Gidaspov  wrote:

> Thank you. Now I'm able to edit tickets. One more question - how can I
> create new version to move some tickets to 2.12?
>
> On 2021/06/04 11:37:55, Pavel Tupitsyn  wrote:
> > Added you to the Contributors role, now you should be able to edit
> tickets.
> >
> >
> > On Fri, Jun 4, 2021 at 2:16 PM Alexey Gidaspov 
> wrote:
> >
> > > Hi, Pavel.
> > >
> > > My JIRA login is agidaspov
> > >
> > > On 2021/06/04 11:14:02, Pavel Tupitsyn  wrote:
> > > > Hi Alexey,
> > > >
> > > > What's your JIRA username?
> > > >
> > > > On Fri, Jun 4, 2021 at 1:50 PM Alexey Gidaspov  >
> > > wrote:
> > > >
> > > > > Hi, All!
> > > > >
> > > > > What should I do to be able to edit fixVersion in jira tickets?
> > > > >
> > > >
> > >
> >
>

Re: Unable to edit fixVersion in jira tickets

[Pavel Tupitsyn]

Added you to the Contributors role, now you should be able to edit tickets.


On Fri, Jun 4, 2021 at 2:16 PM Alexey Gidaspov  wrote:

> Hi, Pavel.
>
> My JIRA login is agidaspov
>
> On 2021/06/04 11:14:02, Pavel Tupitsyn  wrote:
> > Hi Alexey,
> >
> > What's your JIRA username?
> >
> > On Fri, Jun 4, 2021 at 1:50 PM Alexey Gidaspov 
> wrote:
> >
> > > Hi, All!
> > >
> > > What should I do to be able to edit fixVersion in jira tickets?
> > >
> >
>

Re: Unable to edit fixVersion in jira tickets

[Pavel Tupitsyn]

Hi Alexey,

What's your JIRA username?

On Fri, Jun 4, 2021 at 1:50 PM Alexey Gidaspov  wrote:

> Hi, All!
>
> What should I do to be able to edit fixVersion in jira tickets?
>

Re: [MTCGA]: new failures in builds [6024739] needs to be handled

[Pavel Tupitsyn]

Fixed
https://issues.apache.org/jira/browse/IGNITE-14804

On Fri, May 28, 2021 at 5:07 PM Pavel Tupitsyn  wrote:

> Ivan, yes, I'll deal with this next week.
>
> On Fri, May 28, 2021 at 12:28 PM Ivan Daschinsky 
> wrote:
>
>> Hi, Pavel, could you please look at this [1]? It seems after updating
>> TC, few new inspections arrived.
>>
>>
>> ]1] --
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetInspections/6024739?buildTab=Inspection
>>
>> пт, 28 мая 2021 г. в 04:55, :
>> >
>> > Hi Igniters,
>> >
>> >  I've detected some new issue on TeamCity to be handled. You are more
>> than welcomed to help.
>> >
>> >  If your changes can lead to this failure(s): We're grateful that you
>> were a volunteer to make the contribution to this project, but things
>> change and you may no longer be able to finalize your contribution.
>> >  Could you respond to this email and indicate if you wish to continue
>> and fix test failures or step down and some committer may revert you commit.
>> >
>> >  *New Critical Failure in master Platform .NET (Inspections)*
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetInspections?branch=%3Cdefault%3E
>> >  Changes may lead to failure were done by
>> >  - sergei ryzhov 
>> https://ci.ignite.apache.org/viewModification.html?modId=926866
>> >
>> >  - Here's a reminder of what contributors were agreed to do
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>> >  - Should you have any questions please contact
>> dev@ignite.apache.org
>> >
>> > Best Regards,
>> > Apache Ignite TeamCity Bot
>> > https://github.com/apache/ignite-teamcity-bot
>> > Notification generated at 04:55:20 28-05-2021
>>
>

Re: IEP-68: Thin Client Data Streamer

[Pavel Tupitsyn]

Changes are ready for review:

https://issues.apache.org/jira/browse/IGNITE-14187
https://github.com/apache/ignite/pull/8847

On Tue, May 18, 2021 at 12:03 PM Igor Sapego  wrote:

> Pavel,
>
> +1 from me for a stateless approach. We could definitely invent something
> here to make a stateful approach working but that would make this feature
> too complex and error-prone, IMO. It makes sense to file an improvement
> ticket with benchmark results and maybe code draft if we decide to move
> this way in future though.
>
> Best Regards,
> Igor
>
>
> On Tue, May 18, 2021 at 11:47 AM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > I've implemented a "stateful" approach (see IEP) in .NET Thin Client:
> > client-side flush adds data to server-side streamer, but does not flush
> it
> > immediately.
> > It provides ~35% perf improvement over stateless approach (where every
> > client request opens a new streamer, adds data, closes streamer).
> >
> > However, in this case we lose server-side buffered data if the server
> node
> > fails, which seems to be a serious issue.
> > On the client we don't even know which part of data was lost, so we can't
> > re-send this data.
> >
> > Therefore, a stateless approach seems to be the only proper way:
> > once a client request completes, the data is guaranteed to be in the
> cache,
> > and failed requests (connection loss, node failure) can be retried.
> > This simplifies both client and server implementations.
> >
> > Thoughts?
> >
> > On Tue, Mar 9, 2021 at 6:32 PM Pavel Tupitsyn 
> > wrote:
> >
> > > Alex,
> > >
> > > I did not include this for two reasons:
> > >
> > > 1) Existing thick API behavior is very confusing and counter-intuitive:
> > > it returns a Future that won't complete unless you call more APIs.
> > >
> > > I've seen multiple users doing this:
> > > await streamer.Add(1, 1);
> > > await streamer.Add(2, 2);
> > >
> > > This code hangs - the first Add will never complete, because the batch
> is
> > > not sent,
> > > and the batch won't be sent because we are awaiting the first Add.
> > >
> > > 2) I'm not sure what's the use case for this feature - why do we want
> to
> > > know
> > > when the flush happens?
> > >
> > > We can come up with a better API, but should we bother at all?
> > >
> > > On Fri, Mar 5, 2021 at 6:09 PM Alex Plehanov 
> > > wrote:
> > >
> > >> Pavel,
> > >>
> > >> IEP looks good to me overall. I have only one concern: currently, for
> > >> thick
> > >> client "add" method we return the future which completes when the
> > current
> > >> batch is actually added to the cache, even if "flush" method is not
> > >> invoked
> > >> explicitly. For thin client with the proposed protocol we can't
> provide
> > >> such functionality without explicit "flush". Perhaps we should notify
> > >> client when batch actually flushed and send this notification when
> some
> > >> "require notification" flag is sent with the request. WDYT?
> > >>
> > >> пт, 5 мар. 2021 г. в 17:03, Ivan Daschinsky :
> > >>
> > >> > Agree, this is completely offtopic here.
> > >> >
> > >> > пт, 5 мар. 2021 г. в 17:02, Pavel Tupitsyn :
> > >> >
> > >> > > > custom DSL
> > >> > > This is a topic for 3.0 - would you like to start a separate
> thread?
> > >> > >
> > >> > > On Fri, Mar 5, 2021 at 4:54 PM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > I suppose, that the best variantl -- custom DSL similar to
> MongoDB
> > >> > query
> > >> > > > language.
> > >> > > > I understand that implementing this is much harder, but users
> want
> > >> this
> > >> > > > variant.
> > >> > > >
> > >> > > > пт, 5 мар. 2021 г. в 16:52, Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > >> > > >
> > >> > > > > > I suppose this is of questional usability, same for existing
> > >> > > > > > implementation for ScanQuery and ContinousQuery
> > >>

Re: [MTCGA]: new failures in builds [6024739] needs to be handled

[Pavel Tupitsyn]

Ivan, yes, I'll deal with this next week.

On Fri, May 28, 2021 at 12:28 PM Ivan Daschinsky 
wrote:

> Hi, Pavel, could you please look at this [1]? It seems after updating
> TC, few new inspections arrived.
>
>
> ]1] --
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetInspections/6024739?buildTab=Inspection
>
> пт, 28 мая 2021 г. в 04:55, :
> >
> > Hi Igniters,
> >
> >  I've detected some new issue on TeamCity to be handled. You are more
> than welcomed to help.
> >
> >  If your changes can lead to this failure(s): We're grateful that you
> were a volunteer to make the contribution to this project, but things
> change and you may no longer be able to finalize your contribution.
> >  Could you respond to this email and indicate if you wish to continue
> and fix test failures or step down and some committer may revert you commit.
> >
> >  *New Critical Failure in master Platform .NET (Inspections)*
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetInspections?branch=%3Cdefault%3E
> >  Changes may lead to failure were done by
> >  - sergei ryzhov 
> https://ci.ignite.apache.org/viewModification.html?modId=926866
> >
> >  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> >  - Should you have any questions please contact
> dev@ignite.apache.org
> >
> > Best Regards,
> > Apache Ignite TeamCity Bot
> > https://github.com/apache/ignite-teamcity-bot
> > Notification generated at 04:55:20 28-05-2021
>

Re: AggregateUnionTransposeRule fails when some inputs have unique grouping key

[Pavel Tupitsyn]

Hi Vladimir,

Looks like this message is for d...@calcite.apache.org, not
dev@ignite.apache.org,
or am I mistaken?

On Wed, May 19, 2021 at 2:25 PM Vladimir Ozerov  wrote:

> Hi,
>
> The AggregateUnionTransposeRule attempts to push the Aggregate below the
> Union.
>
> Before:
> Aggregate[group=$0, agg=SUM($1]
>   Union[all]
> Input1
> Input2
>
> After:
> Aggregate[group=$0, agg=SUM($1]
>   Union[all]
> Aggregate[group=$0, agg=SUM($1]
>   Input1
> Aggregate[group=$0, agg=SUM($1]
>   Input2
>
> When pushing the Aggregate, it checks whether the input is definitively
> unique on the grouping key. If yes, the Aggregate is not installed on top
> of the input, assuming that the result would be the same as without the
> aggregate. This generates a type mismatch exception when aggregation is
> pushed only to some of the inputs:
> Aggregate[group=$0, agg=SUM($1]
>   Union[all]
> Aggregate[group=$0, agg=SUM($1]
>   Input1
> Input2
>
> It seems that the uniqueness check should not be in that rule at all, and
> the aggregate should be pushed unconditionally. Motivation: we already have
> AggregateRemoveRule that removes unnecessary aggregates. No need to
> duplicate the same non-trivial logic twice.
>
> Does the proposal make sense to you?
>
> Regards,
> Vladimir.
>

Re: IEP-68: Thin Client Data Streamer

[Pavel Tupitsyn]

Igniters,

I've implemented a "stateful" approach (see IEP) in .NET Thin Client:
client-side flush adds data to server-side streamer, but does not flush it
immediately.
It provides ~35% perf improvement over stateless approach (where every
client request opens a new streamer, adds data, closes streamer).

However, in this case we lose server-side buffered data if the server node
fails, which seems to be a serious issue.
On the client we don't even know which part of data was lost, so we can't
re-send this data.

Therefore, a stateless approach seems to be the only proper way:
once a client request completes, the data is guaranteed to be in the cache,
and failed requests (connection loss, node failure) can be retried.
This simplifies both client and server implementations.

Thoughts?

On Tue, Mar 9, 2021 at 6:32 PM Pavel Tupitsyn  wrote:

> Alex,
>
> I did not include this for two reasons:
>
> 1) Existing thick API behavior is very confusing and counter-intuitive:
> it returns a Future that won't complete unless you call more APIs.
>
> I've seen multiple users doing this:
> await streamer.Add(1, 1);
> await streamer.Add(2, 2);
>
> This code hangs - the first Add will never complete, because the batch is
> not sent,
> and the batch won't be sent because we are awaiting the first Add.
>
> 2) I'm not sure what's the use case for this feature - why do we want to
> know
> when the flush happens?
>
> We can come up with a better API, but should we bother at all?
>
> On Fri, Mar 5, 2021 at 6:09 PM Alex Plehanov 
> wrote:
>
>> Pavel,
>>
>> IEP looks good to me overall. I have only one concern: currently, for
>> thick
>> client "add" method we return the future which completes when the current
>> batch is actually added to the cache, even if "flush" method is not
>> invoked
>> explicitly. For thin client with the proposed protocol we can't provide
>> such functionality without explicit "flush". Perhaps we should notify
>> client when batch actually flushed and send this notification when some
>> "require notification" flag is sent with the request. WDYT?
>>
>> пт, 5 мар. 2021 г. в 17:03, Ivan Daschinsky :
>>
>> > Agree, this is completely offtopic here.
>> >
>> > пт, 5 мар. 2021 г. в 17:02, Pavel Tupitsyn :
>> >
>> > > > custom DSL
>> > > This is a topic for 3.0 - would you like to start a separate thread?
>> > >
>> > > On Fri, Mar 5, 2021 at 4:54 PM Ivan Daschinsky 
>> > > wrote:
>> > >
>> > > > I suppose, that the best variantl -- custom DSL similar to MongoDB
>> > query
>> > > > language.
>> > > > I understand that implementing this is much harder, but users want
>> this
>> > > > variant.
>> > > >
>> > > > пт, 5 мар. 2021 г. в 16:52, Pavel Tupitsyn :
>> > > >
>> > > > > > I suppose this is of questional usability, same for existing
>> > > > > > implementation for ScanQuery and ContinousQuery
>> > > > >
>> > > > > One way or another, we need to be able to run custom logic
>> > > > > on the server side from thin clients.
>> > > > >
>> > > > > Our current direction can be seen as "Java is our DSL":
>> > > > > write server-side logic in Java, deploy to servers, call from thin
>> > > > clients.
>> > > > >
>> > > > > This approach is already used for Services, Compute, ScanQuery,
>> > > > > ContinuousQuery.
>> > > > >
>> > > > > If you have a better idea in mind, please share.
>> > > > >
>> > > > > On Fri, Mar 5, 2021 at 4:12 PM Ivan Daschinsky <
>> ivanda...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > >>> - Corresponding types should exist on server nodes
>> > > > > > Ok, but I suppose this is of questional usability, same for
>> > existing
>> > > > > > implementation for ScanQuery and ContinousQuery.
>> > > > > > For queries it's probably ok to add some custom filters and put
>> > them
>> > > in
>> > > > > > servers' classpathes, but I can hardly imagine
>> > > > > > somebody wants to create custom Receiver this way.
>> > > > > >
>> > > > > > пт, 5 мар. 2021 г. в 15