Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan, Val, and others - are there any open objections or questions?
Can we accept the proposal?

On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn  wrote:

> Igniters,
>
> I've updated the IEP to support "Live Schema" [1] from IEP-54.
> Some operations now have schemaless variants, where tuples are serialized
> as maps (String -> val).
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
>
> On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky 
> wrote:
>
>> Val, my understanding about it was exactly the same as yours, but, again,
>> I
>> heard a different opinion.
>>
>> But nevertheless, binary protocol should not be about objects, records aka
>> tuples are the best varii, simple and powerful.
>>
>> As for me, I want to take part in implementing python and golang thin
>> clients for ignite 3, so consider my remarks using this info. I am not
>> just
>> a rude critic,
>> I am just interested in consice and universal binary prorocol
>> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
>> valentin.kuliche...@gmail.com
>> >:
>>
>> > Ivan,
>> >
>> > KV view does work over the tuples. Nested objects and arbitrary
>> structures
>> > can be stored as blobs. So if you need a basic KV cache, you can always
>> > create a table with two blob fields - one for key and one for value -
>> and
>> > store anything there.
>> >
>> > -Val
>> >
>> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky 
>> > wrote:
>> >
>> > > Val, am I right, that kv view over the tuples is just simple mapping
>> from
>> > > POJO to tuple? No collections, no nested objects? I have discussed
>> this
>> > in
>> > > private with Igor and Pavel and they told me different info.
>> > >
>> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
>> > > valentin.kuliche...@gmail.com
>> > > >:
>> > >
>> > > > Ivan,
>> > > >
>> > > > I was answering your question about the KV API. The API I provided
>> has
>> > > been
>> > > > discussed and agreed upon. One of the goals of the protocol is to
>> > > implement
>> > > > this API, so it should give you a clear idea of what we're looking
>> for.
>> > > >
>> > > > Of course, I agree with you that the protocol should be simple and
>> > > flexible
>> > > > enough to allow other implementations for different languages and
>> > > > platforms.
>> > > >
>> > > > -Val
>> > > >
>> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky > >
>> > > > wrote:
>> > > >
>> > > > > Andrey, yep, you are right. This was just a quick idea. As for
>> me, I
>> > > just
>> > > > > don't want to repeat the same problem with compactFooter in thin
>> > client
>> > > > api
>> > > > > of ignite 2.x.
>> > > > >
>> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
>> > andrey.mashen...@gmail.com
>> > > >:
>> > > > >
>> > > > > > >
>> > > > > > > I suppose that we should describe this more verbose and
>> > explicit. I
>> > > > > > > nevertheless suggest to also consider writing values this way:
>> > > > > > > - arr of fields names (if name is missed, corresponding field
>> is
>> > > nil)
>> > > > > > > - arr of rows (row as array, length equal to fields array)
>> > > > > >
>> > > > > >
>> > > > > > Ivan,
>> > > > > > I think GET and PUT operation parameters should be consistent.
>> > > > > > With PUT operation this way may be tricky.
>> > > > > >
>> > > > > > SQL INSERT operation (which is similar PUT operation) semantic
>> > allows
>> > > > > > skipping columns that have a default value.
>> > > > > > Assume we have smth like this:
>> > > > > >
>> > > > > > CREATE TABLE t1 (
>> > > > > >'id' INT;
>> > > > > >'colname' VARCHAR DEFAULT "abc";
>> > > > > > )
>> > > > > > INSERT INTO t1 VALUES(1)
>> > > > > >
>> > > > > > Actually, this will add a row (1, "abc")
>> > > > > >
>> > > > > > Your suggestion related to missed fields will not work this way
>> as
>> > it
>> > > > is
>> > > > > > impossible to distinct
>> > > > > > case with 'null' value from the case with a default value.
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky <
>> > ivanda...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > > Here is the description of TUPLE_GET_ALL:
>> > > > > > > - UUID: table ID
>> > > > > > > - int: schema ID
>> > > > > > > - arr of arr: array of rows with values for all columns in
>> given
>> > > > schema
>> > > > > > >
>> > > > > > > I suppose that we should describe this more verbose and
>> > explicit. I
>> > > > > > > nevertheless suggest to also consider writing values this way:
>> > > > > > > - arr of fields names (if name is missed, corresponding field
>> is
>> > > nil)
>> > > > > > > - arr of rows (row as array, length equal to fields array)
>> > > > > > >
>> > > > > > > It is quite simple and if we use str8 (it is more than enough
>> for
>> > > any
>> > > > > > utf-8
>> > > > > > > reasonable field name), overhead will be negligible, but
>> > > realization
>> > > > > of a
>> > > > > > > client will be way simpl

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Ivan Daschinsky]

Sorry, but what is wrong with simple method isNull()

вт, 6 июл. 2021 г., 09:55 Pavel Tupitsyn :

> Val,
>
> > I don't think there is a significantly better way
> > of doing this in Java.
>
> Yep looks like there is no way to return two values without boxing.
> No ref, no out, no value types.
>
> > Schema already provides this information, doesn't it?
>
> It does, though we don't have an agreement on how to expose this on public
> API yet,
> or do we?
>
> On Tue, Jul 6, 2021 at 12:44 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Pavel,
> >
> > That's a good point, but I don't think there is a significantly better
> way
> > of doing this in Java.
> >
> > There should be a way to check if a field is nullable or not though.
> Schema
> > already provides this information, doesn't it?
> >
> > -Val
> >
> > On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn 
> > wrote:
> >
> > > Igniters,
> > >
> > > Looks like Tuple API has no efficient way to tell if a value for a
> > nullable
> > > column of primitive type is null.
> > >
> > > - Tuple#intValue() will return 0 when the actual value is null => not
> > clear
> > > if 0 is 0 or null.
> > > - Tuple#value() works, but is more expensive due to boxing and type
> > lookup.
> > >
> > > Any ideas on how to improve this?
> > >
> >
>

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Pavel Tupitsyn]

Val,

> I don't think there is a significantly better way
> of doing this in Java.

Yep looks like there is no way to return two values without boxing.
No ref, no out, no value types.

> Schema already provides this information, doesn't it?

It does, though we don't have an agreement on how to expose this on public
API yet,
or do we?

On Tue, Jul 6, 2021 at 12:44 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> That's a good point, but I don't think there is a significantly better way
> of doing this in Java.
>
> There should be a way to check if a field is nullable or not though. Schema
> already provides this information, doesn't it?
>
> -Val
>
> On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > Looks like Tuple API has no efficient way to tell if a value for a
> nullable
> > column of primitive type is null.
> >
> > - Tuple#intValue() will return 0 when the actual value is null => not
> clear
> > if 0 is 0 or null.
> > - Tuple#value() works, but is more expensive due to boxing and type
> lookup.
> >
> > Any ideas on how to improve this?
> >
>

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Valentin Kulichenko]

Pavel,

That's a good point, but I don't think there is a significantly better way
of doing this in Java.

There should be a way to check if a field is nullable or not though. Schema
already provides this information, doesn't it?

-Val

On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn  wrote:

> Igniters,
>
> Looks like Tuple API has no efficient way to tell if a value for a nullable
> column of primitive type is null.
>
> - Tuple#intValue() will return 0 when the actual value is null => not clear
> if 0 is 0 or null.
> - Tuple#value() works, but is more expensive due to boxing and type lookup.
>
> Any ideas on how to improve this?
>

Ignite 3.0 Tuple API: how to check if value is null?

[Pavel Tupitsyn]

Igniters,

Looks like Tuple API has no efficient way to tell if a value for a nullable
column of primitive type is null.

- Tuple#intValue() will return 0 when the actual value is null => not clear
if 0 is 0 or null.
- Tuple#value() works, but is more expensive due to boxing and type lookup.

Any ideas on how to improve this?

Re: Apache Ignite 2.11

[Nikita Safonov]

Hi guys!

Alexey, I do confirm that some of the issues are already resolved (for
example, IGNITE-14658  is
already documented).
What is more, some of the tickets seem to be minor or "very internal"
things that do not change the behaviour/are not accessible for users,
including tests' improvements.

I would say that most of these tickets should be reflected only in the
Release notes, while some should even be omitted.

Alexey, could you or someone else please look through these tickets once
again and say which of them are worth documenting?
Then I'll be able to give the approximate time estimates.

Thank you!

Regards,
Nikita

пн, 5 июл. 2021 г. в 12:20, Alex Plehanov :

> Alexey,
>
> Some of these tickets are already documented. Do we need to clear the "Docs
> Required" flag if documentation for the ticket has been merged to the
> master and release branch? AFAIK we didn't do that for previous releases.
>
> пн, 5 июл. 2021 г. в 13:53, Alexey Gidaspov :
>
> > Hi, Nikita!
> >
> > As I can see, these are the tickets we need to document [1]
> >
> > Can you tell me how long it will take to complete work on documentation?
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.11')%20AND%20cf%5B12313620%5D%20%3D%20%22Docs%20Required%22
> >
> > On 2021/06/09 10:06:09, Nikita Safonov 
> wrote:
> > > Hello Igniters!
> > >
> > > I guess that AI 2.11 will include documentation work.
> > > I'll be glad to help with this.
> > > And I'll be happy to know the scope of features that we need to
> document.
> > >
> > > Regards,
> > > Nikita
> > >
> > > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
> > >
> > > > hello,
> > > >
> > > > can someone can help to merge the following patch into master?
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-12852
> > > >
> > > > This patch has been completed for a long time, and I think it is a
> > > > valuable one.
> > > >
> > > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
> > > > > Sure, thanks for your reply
> > > > >
> > > > > Folks, could you please advice how to setup JIRA account
> integration
> > for
> > > > non-committers?
> > > > >
> > > > > For the page
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
> > > > > I can see all issues, but Alexey's login seems to be not
> sufficient.
> > Can
> > > > wiki admins set up smth? Or Alexey should do some setup on his own?
> > > > >
> > > > > On 2021/06/03 14:55:25, Maksim Timonin 
> > wrote:
> > > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear there
> > are
> > > > >> chances for 2.11, but I guess we can postpone it.
> > > > >>
> > > > >>
> > > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov 
> > > > wrote:
> > > > >>
> > > > >>> ok, Maksim, keep us posted.
> > > > >>>
> > > > >>> We're in the middle of paperwork rigth now, so there is a chance
> ))
> > > > >>>
> > > > >>> On 2021/06/03 14:30:55, Maksim Timonin 
> > > > wrote:
> > > > > Regarding non-SQL index API it really depends on readiness. I
> > guess
> > > > if
> > > > >>> it
> > > >  is not in the master, we can skip and release it next time.
> > > > 
> > > >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release it
> > > > within
> > > > >>> the
> > > >  next release.
> > > > 
> > > >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov <
> dpav...@apache.org>
> > > > wrote:
> > > > 
> > > > > Hi Alexey,
> > > > >
> > > > > Releasing master as-is makes sence for me.
> > > > >
> > > > > Regarding non-SQL index API it really depends on readiness. I
> > guess
> > > > if
> > > > >>> it
> > > > > is not in the master, we can skip and release it next time.
> > > > >
> > > > > We've a bit overdue initial scope freeze date. It's not a big
> > deal,
> > > > but
> > > > > still makes sense to start discussion, create and share wiki
> > page and
> > > > > create branch.
> > > > >
> > > > > Since committer privileges required to create branch, could you
> > > > please
> > > > > notify me in advance.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > On 2021/06/03 14:08:34, ��  <
> > > > > olive.c...@gmail.com> wrote:
> > > > >> :scope:
> > > > >>
> > > > >> Here is a list [1] of 195 tickets, i picked for Apache Ignite
> > 2.11
> > > > > release
> > > > >> Let's start discussion
> > > > >>
> > > > >> [1]
> > > > >>>
> > > >
> >
> https://issues.apache.org/jira/browse/IGNITE-14781?jql=project%20%3D%20IGNITE%20%20AND%20status%20%20%3D%20Resolved%20and%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > >> On 2021/05/13 14:27:33, Dmitry Pavlov 
> > wrote:
> > > > >>> Hi Alexey,
> > > > >>>
> > > > >>> :pmc to sign distribution:
> > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

I've updated the IEP to support "Live Schema" [1] from IEP-54.
Some operations now have schemaless variants, where tuples are serialized
as maps (String -> val).

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)

On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky  wrote:

> Val, my understanding about it was exactly the same as yours, but, again, I
> heard a different opinion.
>
> But nevertheless, binary protocol should not be about objects, records aka
> tuples are the best varii, simple and powerful.
>
> As for me, I want to take part in implementing python and golang thin
> clients for ignite 3, so consider my remarks using this info. I am not just
> a rude critic,
> I am just interested in consice and universal binary prorocol
> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > KV view does work over the tuples. Nested objects and arbitrary
> structures
> > can be stored as blobs. So if you need a basic KV cache, you can always
> > create a table with two blob fields - one for key and one for value - and
> > store anything there.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky 
> > wrote:
> >
> > > Val, am I right, that kv view over the tuples is just simple mapping
> from
> > > POJO to tuple? No collections, no nested objects? I have discussed this
> > in
> > > private with Igor and Pavel and they told me different info.
> > >
> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com
> > > >:
> > >
> > > > Ivan,
> > > >
> > > > I was answering your question about the KV API. The API I provided
> has
> > > been
> > > > discussed and agreed upon. One of the goals of the protocol is to
> > > implement
> > > > this API, so it should give you a clear idea of what we're looking
> for.
> > > >
> > > > Of course, I agree with you that the protocol should be simple and
> > > flexible
> > > > enough to allow other implementations for different languages and
> > > > platforms.
> > > >
> > > > -Val
> > > >
> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > Andrey, yep, you are right. This was just a quick idea. As for me,
> I
> > > just
> > > > > don't want to repeat the same problem with compactFooter in thin
> > client
> > > > api
> > > > > of ignite 2.x.
> > > > >
> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
> > andrey.mashen...@gmail.com
> > > >:
> > > > >
> > > > > > >
> > > > > > > I suppose that we should describe this more verbose and
> > explicit. I
> > > > > > > nevertheless suggest to also consider writing values this way:
> > > > > > > - arr of fields names (if name is missed, corresponding field
> is
> > > nil)
> > > > > > > - arr of rows (row as array, length equal to fields array)
> > > > > >
> > > > > >
> > > > > > Ivan,
> > > > > > I think GET and PUT operation parameters should be consistent.
> > > > > > With PUT operation this way may be tricky.
> > > > > >
> > > > > > SQL INSERT operation (which is similar PUT operation) semantic
> > allows
> > > > > > skipping columns that have a default value.
> > > > > > Assume we have smth like this:
> > > > > >
> > > > > > CREATE TABLE t1 (
> > > > > >'id' INT;
> > > > > >'colname' VARCHAR DEFAULT "abc";
> > > > > > )
> > > > > > INSERT INTO t1 VALUES(1)
> > > > > >
> > > > > > Actually, this will add a row (1, "abc")
> > > > > >
> > > > > > Your suggestion related to missed fields will not work this way
> as
> > it
> > > > is
> > > > > > impossible to distinct
> > > > > > case with 'null' value from the case with a default value.
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > > > - UUID: table ID
> > > > > > > - int: schema ID
> > > > > > > - arr of arr: array of rows with values for all columns in
> given
> > > > schema
> > > > > > >
> > > > > > > I suppose that we should describe this more verbose and
> > explicit. I
> > > > > > > nevertheless suggest to also consider writing values this way:
> > > > > > > - arr of fields names (if name is missed, corresponding field
> is
> > > nil)
> > > > > > > - arr of rows (row as array, length equal to fields array)
> > > > > > >
> > > > > > > It is quite simple and if we use str8 (it is more than enough
> for
> > > any
> > > > > > utf-8
> > > > > > > reasonable field name), overhead will be negligible, but
> > > realization
> > > > > of a
> > > > > > > client will be way simpler
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn  >:
> > > > > > >
> > > > > > > > > No it isn't, I have carefully read code and IEP, in your
> code
> > > you
> > > > > > write
> > > > > > > > > schema id in each tuple.
> > > > > > > >
> > > > > > > > There is no code for batch operations yet.

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Konstantin Orlov]

+1 to Ivan’s idea about adding extra param to IgniteSystemProperties.


> Ok, and what's about other VM arguments, not included to
> IgniteSystemProperties?

IMO all VM arguments should be considered sensitive except those annotated 
with @IgniteSystemProperties(sensitive=false).

-- 
Regards,
Konstantin Orlov




> On 2 Jul 2021, at 20:52, Shishkov Ilya  wrote:
> 
> Ok, and what's about other VM arguments, not included to
> IgniteSystemProperties?
> 
> пт, 2 июл. 2021 г. в 09:27, Zhenya Stanilovsky :
> 
>> 
>> -1 for extra arg, +1 for Ivan`s upper proposal : @IgniteSystemProperty
>> annotation.
>> Look, someone will set some of IGNITE_* option and after possibly cluster
>> problems will give this logs into analysis and engineer can`t reproduce
>> such a case, cause no param is applied.
>> 
>>> An extra argument for IgniteSystemProperty sounds reasonable.
>>> 
>>> -Val
>>> 
>>> On Thu, Jul 1, 2021 at 10:04 AM Ivan Daschinsky < ivanda...@gmail.com >
>> wrote:
>>> 
 Ok, this can be excluded using blocklist-jvm-params.properties or just
>> by
 providing and extra arg to annotation, as I have just suggested
 
 чт, 1 июл. 2021 г., 19:51 Valentin Kulichenko <
 valentin.kuliche...@gmail.com
> :
 
> Ivan,
> 
> IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> (e.g. IGNITE_CONFIG_URL) are often considered sensitive information.
>> Data
> related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely
>> to
 be
> sensitive.
> 
> Once again - I would exclude any property that can contain
>> user-specific
> information. Only our internal settings (stuff
> like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
> 
> -Val
> 
> On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky < ivanda...@gmail.com
>>> 
> wrote:
> 
>> We can add add an extra param in annotation, that blocks param to be
>> printed, just set it to false by default and block it wheb set to
>> true
>> 
>> чт, 1 июл. 2021 г., 19:45 Atri Sharma < a...@apache.org >:
>> 
>>> What if we allowed a blocklist of parameters that are never
>> printed?
>>> 
>>> On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
>>> valentin.kuliche...@gmail.com > wrote:
>>> 
 Not all of them are OK to be printed out. At the very least, we
> should
>>> have
 a mechanism to exclude some of them. I would still go with
>> opt-in
>> rather
 than opt-out though, but I guess that is up to a discussion.
 
 -Val
 
 On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky <
 ivanda...@gmail.com >
 wrote:
 
> This is security through obscurity, an obvious and a
>> well-known
> anti
> pattern. I suppose that printing jvm options, that is
>> registered
 by
> @IgniteSystemProperty annotation is an ideal variant
> 
> чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
>> :
> 
>> Folks,
>> 
>> *Anything* that a user provides to the system can
>> potentially
 be
> considered
>> sensitive information. This includes the VM arguments. We
>> can't
>>> predict
>> what exactly one can put there, so let's not make
>> assumptions.
>> 
>> When dealing with security, we should be as conservative as
>> possible.
> That
>> said, I do not even agree with the pattern approach - there
 might
>> be
>>> a
>> user's system property named IGNITE_xxx. It is also possible
 for
>> our
>> internal properties to contain sensitive information (not
>> all
 of
>> them
 are
>> boolean flags).
>> 
>> The only option I see is to print out specific properties
>> for
> which
>>> we
>> agree that they are safe. For example, we can introduce an
>> annotation
> that
>> would mark "safe" properties in IgniteSystemProperties; we
>> will
>> then
> print
>> out only those that are marked with the annotation.
>> 
>> -Val
>> 
>> 
>> On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> slava.kopti...@gmail.com
>>> 
>> wrote:
>> 
>>> Hello Ivan,
>>> 
 At least, we could just hide params that match a
>> specific
>> pattern
>>> Yes, we can filter out all vm options that do not relate
>> to
>> Ignite,
 for
>>> instance.
>>> 
 Ilya, go ahead, file ticket and prepare a PR.
>>> Please do not rush. Let's listen to other community
>> members.
> This
>> question
>>> is about security and it should not be discussed in a
>> hurry
> (even
> though
>> it
>>> looks like an obvious thing).

Re: IGNITE-14812: Statistics

[Nikita Safonov]

Hello Igniters,

I'd be glad to help with documenting this functionality.
I do believe that it won't take long. Hope to finish it before the end of
next week to have enough time for docs discussion.

Regards,
Nikita

ср, 23 июн. 2021 г. в 18:11, Ivan Daschinsky :

> AFAIK, it is enough to include mention of this library in NOTICE, please
> see here [1][2][3]
>
> [1] -- https://www.apache.org/legal/src-headers.html#3party
> [2] -- https://www.apache.org/legal/src-headers.html#notice
> [3] -- https://www.apache.org/legal/resolved.html
>
> ср, 23 июн. 2021 г. в 17:36, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Taras,
> >
> > It is true that we try to minimize dependencies, but there are some
> anyway.
> > I think it's perfectly fine to add this library as a dependency.
> >
> > Will you be able to do this asap? It is surely better than reverting the
> > commit :)
> >
> > -Val
> >
> > On Wed, Jun 23, 2021 at 12:32 AM Taras Ledkov 
> > wrote:
> >
> > > Hi,
> > >
> > >  > Taras, can you, please, describe the features that was implemented
> in
> > > this merge?
> > >  > How users supposed to use them?
> > >  > Do we have plans to document?
> > >
> > > Sure. Alexander Belyak will describe and file ticket to documentation.
> > >
> > > On 23.06.2021 9:27, Nikolay Izhikov wrote:
> > > > Hello, Taras.
> > > >
> > > > Thanks for feedback.
> > > >
> > > >> AFAIK and as long as I can remember Ignite project try to minimize
> > > external dependencies usage and adds new external dependency only when
> > > there is no other way out.
> > > > Does it mean we have to incapsulate every external library we want to
> > > use?
> > > >
> > > > Taras, can you, please, describe the features that was implemented in
> > > this merge?
> > > > How users supposed to use them?
> > > > Do we have plans to document?
> > > >
> > > >
> > > >> 23 июня 2021 г., в 09:21, Taras Ledkov 
> > > написал(а):
> > > >>
> > > >> Hi,
> > > >>
> > > >> We have discussed BCrypt include/add dependency here [1].
> > > >> AFAIK and as long as I can remember Ignite project try to minimize
> > > external dependencies usage
> > > >> and adds new external dependency only when there is no other way
> out.
> > > >>
> > > >> [1].
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Username-password-authentication-for-thin-clients-tp26058p26954.html
> > > >>
> > > >> On 23.06.2021 3:08, Valentin Kulichenko wrote:
> > > >>> Dmitry,
> > > >>>
> > > >>> As the PMC chair, would you mind contacting legal regarding the
> > matter?
> > > >>> This is not the only example of such code (e.g. [1]), so we should
> > look
> > > >>> into this asap.
> > > >>>
> > > >>> [1]
> > > >>>
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/mindrot/BCrypt.java
> > > >>>
> > > >>> As for this particular commit, can HLL be added as a dependency
> > > instead? Is
> > > >>> there any particular reason to include the source code? @Sasha
> Belyak
> > > >>>  , can you please chime in?
> > > >>>
> > > >>> -Val
> > > >>>
> > > >>> On Tue, Jun 22, 2021 at 8:10 AM Dmitry Pavlov 
> > > wrote:
> > > >>>
> > >  Hi Nikolai,
> > > 
> > >  thank you for noticing. I guess it's not about license, but about
> > >  Intellectual Property (IP) ownership.
> > > 
> > >  AFAIK, Apache License 2.0 is here and AL 2.0 is definetely allowed
> > to
> > > be
> > >  used in the codebase for an Apache project (
> > >  https://www.apache.org/legal/resolved.html)
> > > 
> > >  But licenese and IP owner are 2 sligthly different things. E.g at
> > the
> > > end
> > >  of any website you can find:
> > >  Copyright © 2021 The Apache Software Foundation, Licensed under
> the
> > > Apache
> > >  License, Version 2.0.
> > > 
> > >  Incubated projects are mandated to transfer IP to the ASF. And I'm
> > not
> > >  aware of any exceptions.
> > > 
> > >  In this PR there are 5 classes which licenses with AL 2.0, but IP
> > > owner is
> > >  3rd party company.
> > > 
> > >  I'm a bit concerned about having such code in the project. I'd
> > rather
> > >  reverted it until we have approval from experts at mailing list:
> > >  legal-disc...@apache.org
> > > 
> > >  Sincerely,
> > >  Dmitriy Pavlov
> > > 
> > >  On 2021/06/22 14:56:49, Nikolay Izhikov 
> > wrote:
> > > > Hello, Igniters.
> > > >
> > > > Recently huge commit was merged [1].
> > > >
> > > > Taras, Alexander, can you, please, explain what is purpose of the
> > > commit?
> > > > What feature it implemented?
> > > >
> > > > Looked inside the ticket and found no explanation.
> > > > Description is "Add statistics collection and usage.»
> > > >
> > > > Do we have plans to document this new feature?
> > > >
> > > > Also, I see very strange license in added files [2]
> > > >
> > > > ```
> > > >   * Copyright 2013 Aggregate Knowledge, Inc.
> > 

Re: Apache Ignite 2.11

[Alex Plehanov]

Alexey,

Some of these tickets are already documented. Do we need to clear the "Docs
Required" flag if documentation for the ticket has been merged to the
master and release branch? AFAIK we didn't do that for previous releases.

пн, 5 июл. 2021 г. в 13:53, Alexey Gidaspov :

> Hi, Nikita!
>
> As I can see, these are the tickets we need to document [1]
>
> Can you tell me how long it will take to complete work on documentation?
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.11')%20AND%20cf%5B12313620%5D%20%3D%20%22Docs%20Required%22
>
> On 2021/06/09 10:06:09, Nikita Safonov  wrote:
> > Hello Igniters!
> >
> > I guess that AI 2.11 will include documentation work.
> > I'll be glad to help with this.
> > And I'll be happy to know the scope of features that we need to document.
> >
> > Regards,
> > Nikita
> >
> > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
> >
> > > hello,
> > >
> > > can someone can help to merge the following patch into master?
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-12852
> > >
> > > This patch has been completed for a long time, and I think it is a
> > > valuable one.
> > >
> > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
> > > > Sure, thanks for your reply
> > > >
> > > > Folks, could you please advice how to setup JIRA account integration
> for
> > > non-committers?
> > > >
> > > > For the page
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
> > > > I can see all issues, but Alexey's login seems to be not sufficient.
> Can
> > > wiki admins set up smth? Or Alexey should do some setup on his own?
> > > >
> > > > On 2021/06/03 14:55:25, Maksim Timonin 
> wrote:
> > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear there
> are
> > > >> chances for 2.11, but I guess we can postpone it.
> > > >>
> > > >>
> > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov 
> > > wrote:
> > > >>
> > > >>> ok, Maksim, keep us posted.
> > > >>>
> > > >>> We're in the middle of paperwork rigth now, so there is a chance ))
> > > >>>
> > > >>> On 2021/06/03 14:30:55, Maksim Timonin 
> > > wrote:
> > > > Regarding non-SQL index API it really depends on readiness. I
> guess
> > > if
> > > >>> it
> > >  is not in the master, we can skip and release it next time.
> > > 
> > >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release it
> > > within
> > > >>> the
> > >  next release.
> > > 
> > >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov 
> > > wrote:
> > > 
> > > > Hi Alexey,
> > > >
> > > > Releasing master as-is makes sence for me.
> > > >
> > > > Regarding non-SQL index API it really depends on readiness. I
> guess
> > > if
> > > >>> it
> > > > is not in the master, we can skip and release it next time.
> > > >
> > > > We've a bit overdue initial scope freeze date. It's not a big
> deal,
> > > but
> > > > still makes sense to start discussion, create and share wiki
> page and
> > > > create branch.
> > > >
> > > > Since committer privileges required to create branch, could you
> > > please
> > > > notify me in advance.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > On 2021/06/03 14:08:34, ��  <
> > > > olive.c...@gmail.com> wrote:
> > > >> :scope:
> > > >>
> > > >> Here is a list [1] of 195 tickets, i picked for Apache Ignite
> 2.11
> > > > release
> > > >> Let's start discussion
> > > >>
> > > >> [1]
> > > >>>
> > >
> https://issues.apache.org/jira/browse/IGNITE-14781?jql=project%20%3D%20IGNITE%20%20AND%20status%20%20%3D%20Resolved%20and%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > >> On 2021/05/13 14:27:33, Dmitry Pavlov 
> wrote:
> > > >>> Hi Alexey,
> > > >>>
> > > >>> :pmc to sign distribution:
> > > >>> Formally, only PMC member can be a release manager, so part of
> > > > activities should be picked up by committer and PMC member. If
> noone
> > > >>> else
> > > > want to help you here, I would.
> > > >>> :timeline:
> > > >>> And, could you estimate all phases of the release.
> > > >>>
> > > >>> :scope:
> > > >>> I guess we can just pick up master current state and release
> it.
> > > >>> But
> > > > if someone has some ideas if we should wait for particular
> feature to
> > > >>> be
> > > > completed before branch divergence/release candidate build,
> please
> > > let
> > > >>> know.
> > > >>> Sincerely,
> > > >>> Dmitriy Pavlov
> > > >>>
> > > >>> On 2021/05/13 09:45:02, Алексей Гидаспов  >
> > > > wrote:
> > >  Dear Ignite Community!
> > > 
> > >  I suggest starting Apache Ignite 2.11 release activities. Also
> > > > suggest
> > >  myself to be a release manager for this release. I plan
> rele

Re: Apache Ignite 2.11

[Alexey Gidaspov]

Hi, Nikita!

As I can see, these are the tickets we need to document [1]

Can you tell me how long it will take to complete work on documentation?

[1] 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.11')%20AND%20cf%5B12313620%5D%20%3D%20%22Docs%20Required%22

On 2021/06/09 10:06:09, Nikita Safonov  wrote: 
> Hello Igniters!
> 
> I guess that AI 2.11 will include documentation work.
> I'll be glad to help with this.
> And I'll be happy to know the scope of features that we need to document.
> 
> Regards,
> Nikita
> 
> пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
> 
> > hello,
> >
> > can someone can help to merge the following patch into master?
> >
> > https://issues.apache.org/jira/browse/IGNITE-12852
> >
> > This patch has been completed for a long time, and I think it is a
> > valuable one.
> >
> > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
> > > Sure, thanks for your reply
> > >
> > > Folks, could you please advice how to setup JIRA account integration for
> > non-committers?
> > >
> > > For the page
> > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
> > > I can see all issues, but Alexey's login seems to be not sufficient. Can
> > wiki admins set up smth? Or Alexey should do some setup on his own?
> > >
> > > On 2021/06/03 14:55:25, Maksim Timonin  wrote:
> > >> I mean I'm OK to release it with 2.12. So, I am glad to hear there are
> > >> chances for 2.11, but I guess we can postpone it.
> > >>
> > >>
> > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov 
> > wrote:
> > >>
> > >>> ok, Maksim, keep us posted.
> > >>>
> > >>> We're in the middle of paperwork rigth now, so there is a chance ))
> > >>>
> > >>> On 2021/06/03 14:30:55, Maksim Timonin 
> > wrote:
> > > Regarding non-SQL index API it really depends on readiness. I guess
> > if
> > >>> it
> >  is not in the master, we can skip and release it next time.
> > 
> >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release it
> > within
> > >>> the
> >  next release.
> > 
> >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov 
> > wrote:
> > 
> > > Hi Alexey,
> > >
> > > Releasing master as-is makes sence for me.
> > >
> > > Regarding non-SQL index API it really depends on readiness. I guess
> > if
> > >>> it
> > > is not in the master, we can skip and release it next time.
> > >
> > > We've a bit overdue initial scope freeze date. It's not a big deal,
> > but
> > > still makes sense to start discussion, create and share wiki page and
> > > create branch.
> > >
> > > Since committer privileges required to create branch, could you
> > please
> > > notify me in advance.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > On 2021/06/03 14:08:34, ��  <
> > > olive.c...@gmail.com> wrote:
> > >> :scope:
> > >>
> > >> Here is a list [1] of 195 tickets, i picked for Apache Ignite 2.11
> > > release
> > >> Let's start discussion
> > >>
> > >> [1]
> > >>>
> > https://issues.apache.org/jira/browse/IGNITE-14781?jql=project%20%3D%20IGNITE%20%20AND%20status%20%20%3D%20Resolved%20and%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > >> On 2021/05/13 14:27:33, Dmitry Pavlov  wrote:
> > >>> Hi Alexey,
> > >>>
> > >>> :pmc to sign distribution:
> > >>> Formally, only PMC member can be a release manager, so part of
> > > activities should be picked up by committer and PMC member. If noone
> > >>> else
> > > want to help you here, I would.
> > >>> :timeline:
> > >>> And, could you estimate all phases of the release.
> > >>>
> > >>> :scope:
> > >>> I guess we can just pick up master current state and release it.
> > >>> But
> > > if someone has some ideas if we should wait for particular feature to
> > >>> be
> > > completed before branch divergence/release candidate build, please
> > let
> > >>> know.
> > >>> Sincerely,
> > >>> Dmitriy Pavlov
> > >>>
> > >>> On 2021/05/13 09:45:02, Алексей Гидаспов 
> > > wrote:
> >  Dear Ignite Community!
> > 
> >  I suggest starting Apache Ignite 2.11 release activities. Also
> > > suggest
> >  myself to be a release manager for this release. I plan release
> > >>> at
> > > the end
> >  of june 2021.
> > 
> >  ___
> > 
> >  Best Regards,
> >  Alexey Gidaspov
> > 
> >
> 

flaky test SqlSystemViewsSelfTest.testCachesViews

[Sergei Ryzhov]

Hello, Igniters.

There is a PR for this problem
https://issues.apache.org/jira/browse/IGNITE-15042
https://github.com/apache/ignite/pull/9215

test failed due to affinity.toString comparison
Contains a lot of unnecessary fields.

-- 
Best regards,
Sergei Ryzhov

MOVING Partitions and Rebalancing

[Atri Sharma]

Hi All,

As part of the text queries overhaul effort, I am looking into the
following ticket:

https://issues.apache.org/jira/browse/IGNITE-12401

As I understand it, the problem lies in the fact that a partition can
move, thus causing duplicate data between two Lucene indices (on
different nodes).

I wish to discuss the solution to this problem:

1. Fix the issue with MOVING partitions (need help here, since I am
not aware of the internals of rebalancing).

2. Circumvent the problem for text queries (maybe assign/use a
globally unique ID per entry, and use that to remove duplicates during
the merge phase?)

Please share thoughts and inputs.

Atri