Re: Text Queries Support

[Ivan Pavlukhin]

Hi Atri,

My main concern is non-maleficence. Every task has several solutions,
e.g. straightforward ones:
1. Do not implement FTS.
2. Create own implementation.

Some of the strongest ones live without FTS [1].

[1] https://github.com/cockroachdb/cockroach/issues/7821

2021-08-02 11:33 GMT+03:00, Atri Sharma :
> Hi Ivan,
>
> Would you like to propose an alternative to Lucene?
>
> Atri
>
> On Mon, 2 Aug 2021, 13:48 Ivan Pavlukhin,  wrote:
>
>> Folks,
>>
>> Sorry if read the thread not thoroughly enough, but do we consider
>> Lucene as obviously right choice? In my understanding Ignite history
>> has shown clearly that "fastest feature implementation" is not usually
>> the best. And one example of this are text queries. Are not we trying
>> to do a same mistake again? FTS is a huge feature, I do not believe
>> there is an easy win for it.
>>
>> 2021-07-27 19:18 GMT+03:00, Atri Sharma :
>> > Andrey,
>> >
>> >> Per-partition Lucene index looks simple to implement, but it may
>> >> require
>> >> per-partition SQL to make full-text search expressions work correctly
>> >> within the SQL quiery.
>> > I think that as long as we follow the map - reduce process that we
>> > already do for other queries, we should be fine.
>> >
>> >> Per-partition SQL index may kill the performance. We already tried to
>> >> do
>> >> that in Ignite 2. However, QueryParallelism feature helps to speed up
>> >> some
>> >> data-intensive queries,
>> >> but hits the performance in simple cases, and at some point (e.g.
>> >> segments
>> >> > number of CPU) the performance rapidly degrades with the increasing
>> >> number of segments.
>> >
>> > Yeah, that is always the case, but a global index will be a nightmare
>> > in terms of concurrency and pessimistic concurrency control will
>> > anyways kill the benefits, coupled with the metadata requirements.
>> > What were the specific issues with per partition index?
>> >>
>> >> AFAIK, Lucene widely used bitmap indices that are easy to merge.
>> >> Maybe, the map-reduce technique underneath FTS expressions and some
>> hacks
>> >> will add a minimal overhead.
>> >
>> > Lucene uses many types of indices but the aspect here is that per
>> > partition Lucene indices can return docIDs and we can merge them in
>> > reduce phase. So we are abstracted out from specifics of the internal
>> > index being used to serve the query.
>> >
>> >>
>> >> > As illustrated by Ilya, we can use Ignite's WAL records to rebuild
>> >> > Lucene indices. The important thing here is to not treat Lucene
>> >> > indices as source of truth.
>> >> To use WAL we either should relay Lucene files to our Page memory or
>> >> be
>> >> aware of Lucene files structure.
>> >> The first looks tricky, as we should guarantee a contiguous address
>> space
>> >> in Page memory for reflecting Lucene file. Maybe separate managed
>> >> memory
>> >> segment with its own rules?
>> >
>> > Why not use Lucene's MMappedDirectory and map it to our storage
>> > classes?
>> >
>> >>
>> >> >> Transactions.
>> >> >> * Will we support transactions?
>> >> > Lucene has no concept of transactions.
>> >> Yes, but we have.
>> >> Lucene index may be non-transactional, but users never expect to see
>> >> uncommited data.
>> >> How does this connect with transactional SQL?
>> > We could have the Lucene writes done as a part of transactions and ack
>> > back only when it succeeds/fails. WDYT?
>> >>
>> >> On Tue, Jul 27, 2021 at 1:36 PM Atri Sharma  wrote:
>> >>
>> >> > Sorry, I planned on creating a Wiki page for this, but it makes more
>> >> > sense to be replying here.
>> >> >
>> >> > > * How Lucene index can be split among the nodes?
>> >> >
>> >> > We can have partition level indices on each node.
>> >> >
>> >> > > * If we'll have a single index for all partitions on the
>> >> > > particular
>> >> > > node,
>> >> > > then how index records will be aware of partitioning?
>> >> >
>> >> > Index records dont need to be aware of partitioning -- each Lucene
>> >> > index is independent.
>> >> >
>> >> > > This is important to filter out backup records from the results to
>> >> > > avoid
>> >> > > duplicates.
>> >> >
>> >> > We can merge documents from different nodes and remove duplicates as
>> >> > long as docIDs are globally unique.
>> >> >
>> >> > > * How results from several nodes can be merged on the Reduce
>> >> > > stage?
>> >> >
>> >> > As long as documents have a globally unique docID, Lucene has merge
>> >> > functions that can merge results from multiple partial results.
>> >> >
>> >> > > * Does Lucene supports smth like JOIN operation or others that may
>> >> > require
>> >> > > data from another partition or index?
>> >> >
>> >> > As illustrated by Ilya, Block-Join works for us.
>> >> >
>> >> > > If so, then it likes to multistep query with merging results on
>> >> > > intermediate stages and requires detailed investigation and
>> >> > > design.
>> >> > > It is ok if Ignite will have some limitations here, but we would
>> like
>> >> > > to
>> >> > > know 

Re: Apache Ignite 3 Alpha 2 webinar follow up questions

[Ivan Pavlukhin]

Hi Courtney,

Statistics for query planning is a rather complex subject.
Unfortunately I am now aware of technical details how is it expected
to be implemented in Ignite 3.

Folks driving SQL please step in.

2021-07-31 20:18 GMT+03:00, Courtney Robinson :
> Hi Ivan,
> Atri's description of the query plan being cached is what I was thinking of
> with my description.
>
> I lack the knowledge on how the statistics are maintained to really comment
> constructively Atri but my first question about the problem you raise with
> statistics would be:
>
> How/where are the stats maintained and if a query plan is cached based on
> some stats, is it not possible to invalidate the cached plan periodically
> or based on statistics changes?
>
> Regards,
> Courtney Robinson
> Founder and CEO, Hypi
> Tel: ++44 208 123 2413 (GMT+0) 
>
> 
> https://hypi.io
>
>
> On Sat, Jul 31, 2021 at 8:54 AM Atri Sharma  wrote:
>
>> Query caching works on three levels - caching results, caching blocks and
>> caching query plans.
>>
>> Prepared queries work by caching a plan for a query and reusing that plan
>> by changing the parameters for the incoming query. So the query remains
>> the
>> same, but input values keep changing.
>>
>> The problem with prepared queries is that query execution can go bad very
>> fast if the underlying data distribution changes and the cached plan is
>> no
>> longer optimal for the given statistics.
>>
>> On Sat, 31 Jul 2021, 12:54 Ivan Pavlukhin,  wrote:
>>
>> > Hi Courtney,
>> >
>> > Please clarify what do you mean by prepared queries and query caching?
>> > Do you mean caching query results? If so, in my mind material views
>> > are the best approach here (Ignite 2 does not support them). Do you
>> > have other good approaches in your mind? E.g. implemented in other
>> > databases.
>> >
>> > 2021-07-26 21:27 GMT+03:00, Valentin Kulichenko <
>> > valentin.kuliche...@gmail.com>:
>> > > Hi Courtney,
>> > >
>> > > Generally speaking, query caching certainly makes sense. As far as I
>> > know,
>> > > Ignite 2.x actually does that, but most likely there might be room
>> > > for
>> > > improvement as well. We will look into this.
>> > >
>> > > As for the SQL API - the answer is yes. The requirement for a dummy
>> cache
>> > > is an artifact of the current architecture. This is 100% wrong and
>> > > will
>> > be
>> > > changed in 3.0.
>> > >
>> > > -Val
>> > >
>> > > On Sun, Jul 25, 2021 at 2:51 PM Courtney Robinson
>> > > 
>> > > wrote:
>> > >
>> > >> Something else came to mind, are there plans to support prepared
>> > queries?
>> > >>
>> > >> I recall someone saying before that Ignite does internally cache
>> queries
>> > >> but it's not at all clear if or how it does do that. I assume a
>> > >> simple
>> > >> hash
>> > >> of the query isn't enough.
>> > >>
>> > >> We generate SQL queries based on user runtime settings and they can
>> get
>> > >> to
>> > >> hundreds of lines long, I imagine this means most of our queries are
>> not
>> > >> being cached but there are patterns so we could generate and manage
>> > >> prepared queries ourselves.
>> > >>
>> > >> Also, will there be a dedicated API for doing SQL queries rather
>> > >> than
>> > >> having to pass a SqlFieldsQuery to a cache that has nothing to do
>> > >> with
>> > >> the
>> > >> cache being queried? When I first started with Ignite years ago,
>> > >> this
>> > was
>> > >> beyond confusing for me. I'm trying to run select x from B but I
>> > >> pass
>> > >> this
>> > >> to a cache called DUMMY or whatever arbitrary name...
>> > >>
>> > >> On Fri, Jul 23, 2021 at 4:05 PM Courtney Robinson <
>> > >> courtney.robin...@hypi.io>
>> > >> wrote:
>> > >>
>> > >> > Andrey,
>> > >> > Thanks for the response - see my comments inline.
>> > >> >
>> > >> >
>> > >> >> I've gone through the questions and have no the whole picture of
>> your
>> > >> use
>> > >> >> case.
>> > >> >
>> > >> > Would you please clarify how you exactly use the Ignite? what are
>> the
>> > >> >> integration points?
>> > >> >>
>> > >> >
>> > >> > I'll try to clarify - we have a low/no code platform. A user
>> designs a
>> > >> > model for their application and we map this model to Ignite tables
>> and
>> > >> > other data sources. The model I'll describe is what we're building
>> now
>> > >> and
>> > >> > expected to be in alpha some time in Q4 21. Our current production
>> > >> > architecture is different and isn't as generic, it is heavily tied
>> to
>> > >> > Ignite and we've redesigned to get some flexibility where Ignite
>> > >> > doesn't
>> > >> > provide what we want. Things like window functions and other
>> > >> > SQL-99
>> > >> limits.
>> > >> >
>> > >> > In the next gen version we're working on you can create a model
>> > >> > for
>> a
>> > >> > Tweet(content, to) and we will create an Ignite table with content
>> and
>> > >> > to
>> > >> > columns using the type the user selects. This is the simplest
>> > >> > case.
>> > >> > We are adding generic 

Re: Text Queries Support

[Atri Sharma]

Hi Ivan,

I didn't quite understand. Are you proposing that Ignite should not
have FTS capabilities?

Atri

On Tue, Aug 3, 2021 at 2:57 PM Ivan Pavlukhin  wrote:
>
> Hi Atri,
>
> My main concern is non-maleficence. Every task has several solutions,
> e.g. straightforward ones:
> 1. Do not implement FTS.
> 2. Create own implementation.
>
> Some of the strongest ones live without FTS [1].
>
> [1] https://github.com/cockroachdb/cockroach/issues/7821
>
> 2021-08-02 11:33 GMT+03:00, Atri Sharma :
> > Hi Ivan,
> >
> > Would you like to propose an alternative to Lucene?
> >
> > Atri
> >
> > On Mon, 2 Aug 2021, 13:48 Ivan Pavlukhin,  wrote:
> >
> >> Folks,
> >>
> >> Sorry if read the thread not thoroughly enough, but do we consider
> >> Lucene as obviously right choice? In my understanding Ignite history
> >> has shown clearly that "fastest feature implementation" is not usually
> >> the best. And one example of this are text queries. Are not we trying
> >> to do a same mistake again? FTS is a huge feature, I do not believe
> >> there is an easy win for it.
> >>
> >> 2021-07-27 19:18 GMT+03:00, Atri Sharma :
> >> > Andrey,
> >> >
> >> >> Per-partition Lucene index looks simple to implement, but it may
> >> >> require
> >> >> per-partition SQL to make full-text search expressions work correctly
> >> >> within the SQL quiery.
> >> > I think that as long as we follow the map - reduce process that we
> >> > already do for other queries, we should be fine.
> >> >
> >> >> Per-partition SQL index may kill the performance. We already tried to
> >> >> do
> >> >> that in Ignite 2. However, QueryParallelism feature helps to speed up
> >> >> some
> >> >> data-intensive queries,
> >> >> but hits the performance in simple cases, and at some point (e.g.
> >> >> segments
> >> >> > number of CPU) the performance rapidly degrades with the increasing
> >> >> number of segments.
> >> >
> >> > Yeah, that is always the case, but a global index will be a nightmare
> >> > in terms of concurrency and pessimistic concurrency control will
> >> > anyways kill the benefits, coupled with the metadata requirements.
> >> > What were the specific issues with per partition index?
> >> >>
> >> >> AFAIK, Lucene widely used bitmap indices that are easy to merge.
> >> >> Maybe, the map-reduce technique underneath FTS expressions and some
> >> hacks
> >> >> will add a minimal overhead.
> >> >
> >> > Lucene uses many types of indices but the aspect here is that per
> >> > partition Lucene indices can return docIDs and we can merge them in
> >> > reduce phase. So we are abstracted out from specifics of the internal
> >> > index being used to serve the query.
> >> >
> >> >>
> >> >> > As illustrated by Ilya, we can use Ignite's WAL records to rebuild
> >> >> > Lucene indices. The important thing here is to not treat Lucene
> >> >> > indices as source of truth.
> >> >> To use WAL we either should relay Lucene files to our Page memory or
> >> >> be
> >> >> aware of Lucene files structure.
> >> >> The first looks tricky, as we should guarantee a contiguous address
> >> space
> >> >> in Page memory for reflecting Lucene file. Maybe separate managed
> >> >> memory
> >> >> segment with its own rules?
> >> >
> >> > Why not use Lucene's MMappedDirectory and map it to our storage
> >> > classes?
> >> >
> >> >>
> >> >> >> Transactions.
> >> >> >> * Will we support transactions?
> >> >> > Lucene has no concept of transactions.
> >> >> Yes, but we have.
> >> >> Lucene index may be non-transactional, but users never expect to see
> >> >> uncommited data.
> >> >> How does this connect with transactional SQL?
> >> > We could have the Lucene writes done as a part of transactions and ack
> >> > back only when it succeeds/fails. WDYT?
> >> >>
> >> >> On Tue, Jul 27, 2021 at 1:36 PM Atri Sharma  wrote:
> >> >>
> >> >> > Sorry, I planned on creating a Wiki page for this, but it makes more
> >> >> > sense to be replying here.
> >> >> >
> >> >> > > * How Lucene index can be split among the nodes?
> >> >> >
> >> >> > We can have partition level indices on each node.
> >> >> >
> >> >> > > * If we'll have a single index for all partitions on the
> >> >> > > particular
> >> >> > > node,
> >> >> > > then how index records will be aware of partitioning?
> >> >> >
> >> >> > Index records dont need to be aware of partitioning -- each Lucene
> >> >> > index is independent.
> >> >> >
> >> >> > > This is important to filter out backup records from the results to
> >> >> > > avoid
> >> >> > > duplicates.
> >> >> >
> >> >> > We can merge documents from different nodes and remove duplicates as
> >> >> > long as docIDs are globally unique.
> >> >> >
> >> >> > > * How results from several nodes can be merged on the Reduce
> >> >> > > stage?
> >> >> >
> >> >> > As long as documents have a globally unique docID, Lucene has merge
> >> >> > functions that can merge results from multiple partial results.
> >> >> >
> >> >> > > * Does Lucene supports smth like JOIN operation or others that may
> >> >

Secondary TeamCity instance: ci2.ignite.apache.org

[Dmitry Pavlov]

Hi Igniters,

I'm glad to announce that SberTech made an internal aggreement to sponsor a 
computing power to provide backup TeamCity instance. This instance is intended 
to be a geo-independent, secondary instance with availablility for community 
members. 

Thanks to JetBrains for providing license keys for TeamCity as their part of 
opensource sponsoring program.

Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o. Please 
share your vision what would be most obvious.

After a private discussions with Vitaliy Osipov and Petr Ivanov, I do believe 
that it would be possible to preserve current registered users. Build 
configurations are to be the same for the secondary instance. Technical details 
on how is could be achieved will be available later (most likely we'll start a 
sepearate thread to dicuss that).

You are more than welcome to be an early adopters.

Sincerely,
Dmitriy Pavlov

Re: Secondary TeamCity instance: ci2.ignite.apache.org

[Pavel Tupitsyn]

Dmitry,

This is awesome!

> tc.i.a.o or ci2.i.a.o
My vote for ci2.i.a.o

On Tue, Aug 3, 2021 at 6:33 PM Dmitry Pavlov  wrote:

> Hi Igniters,
>
> I'm glad to announce that SberTech made an internal aggreement to sponsor
> a computing power to provide backup TeamCity instance. This instance is
> intended to be a geo-independent, secondary instance with availablility for
> community members.
>
> Thanks to JetBrains for providing license keys for TeamCity as their part
> of opensource sponsoring program.
>
> Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o. Please
> share your vision what would be most obvious.
>
> After a private discussions with Vitaliy Osipov and Petr Ivanov, I do
> believe that it would be possible to preserve current registered users.
> Build configurations are to be the same for the secondary instance.
> Technical details on how is could be achieved will be available later (most
> likely we'll start a sepearate thread to dicuss that).
>
> You are more than welcome to be an early adopters.
>
> Sincerely,
> Dmitriy Pavlov
>
>

Re: Re[2]: [ANNOUNCE] Welcome Zhenya Stanilovsky as a new committer

[Dmitry Pavlov]

Zhenya, congrats with new role. Well deserved!

On 2021/07/30 16:10:41, Shishkov Ilya  wrote: 
> Zhenya,
> 
> Congrats!
> 
> пт, 30 июл. 2021 г. в 14:01, Zhenya Stanilovsky  >:
> 
> >
> >
> > Guys, thank you very much !!
> >
> > >Zhenya,
> > >
> > >Congrats!
> > >
> > >--
> > >Regards,
> > >Konstantin Orlov
> > >
> > >
> > >
> > >
> > >> On 30 Jul 2021, at 12:20, Вячеслав Коптилин < slava.kopti...@gmail.com
> > > wrote:
> > >>
> > >> Hooray!
> > >>
> > >> Congrats! May the Force be with you!
> > >>
> > >> Thanks,
> > >> S.
> > >>
> > >> пт, 30 июл. 2021 г. в 11:17, Anton Vinogradov < a...@apache.org >:
> > >>
> > >>> Congrats!
> > >>>
> > >>> On Fri, Jul 30, 2021 at 10:19 AM ткаленко кирилл <
> > tkalkir...@yandex.ru >
> > >>> wrote:
> > >>>
> >  Zhenya, congratulations!
> > 
> >   Пересылаемое сообщение 
> >  30.07.2021, 09:50, "Ivan Daschinsky" < ivanda...@apache.org >:
> > 
> > 
> >  Zhenya, congrats, well deserved!
> > 
> >  пт, 30 июл. 2021 г. в 00:44, Andrey Mashenkov <
> > >>>  andrey.mashen...@gmail.com
> > > :
> > 
> > > Congratulations Zhenya!
> > >
> > > On Fri, Jul 30, 2021 at 12:06 AM Maxim Muzafarov < mmu...@apache.org
> > >
> > > wrote:
> > >
> > >> The Project Management Committee (PMC) for Apache Ignite has invited
> > >> Zhenya Stanilovsky to become a committer and we are pleased to
> >  announce
> > >> that
> > >> he has accepted.
> > >>
> > >> For the last few years, Zhenya made a lot of performance fixes
> > >> especially for the core modules and important contributions to the
> > >> Apache Ignite codebase. He is actively involved in integrating the
> > >> Calcite framework with Apache Ignite. Moreover, he has been a great
> > >> help in the preparation of several Apache Ignite major releases by
> > >> carrying out stress-load tests. Besides the code contributions,
> > >>> Zhenya
> > >> is also an active community member and help users on dev and users
> > >> lists.
> > >>
> > >> Being a committer enables easier contribution to the project since
> >  there
> > > is
> > >> no need to go via the patch submission process. This should enable
> >  better
> > >> productivity.
> > >>
> > >> Please join me in welcoming Ivan, and congratulating him on the new
> >  role
> > > in
> > >> the Apache Ignite Community.
> > >>
> > >> Best Regards,
> > >> Maxim Muzafarov
> > >>
> > >
> > > --
> > > Best regards,
> > > Andrey V. Mashenkov
> > 
> >   Конец пересылаемого сообщения 
> > 
> > >>>
> >
> >
> >
> >
> 

Re: Secondary TeamCity instance: ci2.ignite.apache.org

[Petr Ivanov]

Seems that preserving and syncing users will be difficult to achieve — that 
info is stored in DB, and partial DB replication is tricky.

Also ci2.ignite.apache.org is good name, but currently it is targeted the same 
IP as original one:
; ANSWER SECTION:
ci2.ignite.apache.org.  1726IN  A   195.144.253.150



> On 3 Aug 2021, at 18:33, Dmitry Pavlov  wrote:
> 
> Hi Igniters,
> 
> I'm glad to announce that SberTech made an internal aggreement to sponsor a 
> computing power to provide backup TeamCity instance. This instance is 
> intended to be a geo-independent, secondary instance with availablility for 
> community members. 
> 
> Thanks to JetBrains for providing license keys for TeamCity as their part of 
> opensource sponsoring program.
> 
> Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o. Please 
> share your vision what would be most obvious.
> 
> After a private discussions with Vitaliy Osipov and Petr Ivanov, I do believe 
> that it would be possible to preserve current registered users. Build 
> configurations are to be the same for the secondary instance. Technical 
> details on how is could be achieved will be available later (most likely 
> we'll start a sepearate thread to dicuss that).
> 
> You are more than welcome to be an early adopters.
> 
> Sincerely,
> Dmitriy Pavlov
> 

Re: Re[2]: [ANNOUNCE] Welcome Zhenya Stanilovsky as a new committer

[Alexei Scherbakov]

My regards!

вт, 3 авг. 2021 г. в 19:02, Dmitry Pavlov :

> Zhenya, congrats with new role. Well deserved!
>
> On 2021/07/30 16:10:41, Shishkov Ilya  wrote:
> > Zhenya,
> >
> > Congrats!
> >
> > пт, 30 июл. 2021 г. в 14:01, Zhenya Stanilovsky
>  > >:
> >
> > >
> > >
> > > Guys, thank you very much !!
> > >
> > > >Zhenya,
> > > >
> > > >Congrats!
> > > >
> > > >--
> > > >Regards,
> > > >Konstantin Orlov
> > > >
> > > >
> > > >
> > > >
> > > >> On 30 Jul 2021, at 12:20, Вячеслав Коптилин <
> slava.kopti...@gmail.com
> > > > wrote:
> > > >>
> > > >> Hooray!
> > > >>
> > > >> Congrats! May the Force be with you!
> > > >>
> > > >> Thanks,
> > > >> S.
> > > >>
> > > >> пт, 30 июл. 2021 г. в 11:17, Anton Vinogradov < a...@apache.org >:
> > > >>
> > > >>> Congrats!
> > > >>>
> > > >>> On Fri, Jul 30, 2021 at 10:19 AM ткаленко кирилл <
> > > tkalkir...@yandex.ru >
> > > >>> wrote:
> > > >>>
> > >  Zhenya, congratulations!
> > > 
> > >   Пересылаемое сообщение 
> > >  30.07.2021, 09:50, "Ivan Daschinsky" < ivanda...@apache.org >:
> > > 
> > > 
> > >  Zhenya, congrats, well deserved!
> > > 
> > >  пт, 30 июл. 2021 г. в 00:44, Andrey Mashenkov <
> > > >>>  andrey.mashen...@gmail.com
> > > > :
> > > 
> > > > Congratulations Zhenya!
> > > >
> > > > On Fri, Jul 30, 2021 at 12:06 AM Maxim Muzafarov <
> mmu...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> The Project Management Committee (PMC) for Apache Ignite has
> invited
> > > >> Zhenya Stanilovsky to become a committer and we are pleased to
> > >  announce
> > > >> that
> > > >> he has accepted.
> > > >>
> > > >> For the last few years, Zhenya made a lot of performance fixes
> > > >> especially for the core modules and important contributions to
> the
> > > >> Apache Ignite codebase. He is actively involved in integrating
> the
> > > >> Calcite framework with Apache Ignite. Moreover, he has been a
> great
> > > >> help in the preparation of several Apache Ignite major releases
> by
> > > >> carrying out stress-load tests. Besides the code contributions,
> > > >>> Zhenya
> > > >> is also an active community member and help users on dev and
> users
> > > >> lists.
> > > >>
> > > >> Being a committer enables easier contribution to the project
> since
> > >  there
> > > > is
> > > >> no need to go via the patch submission process. This should
> enable
> > >  better
> > > >> productivity.
> > > >>
> > > >> Please join me in welcoming Ivan, and congratulating him on the
> new
> > >  role
> > > > in
> > > >> the Apache Ignite Community.
> > > >>
> > > >> Best Regards,
> > > >> Maxim Muzafarov
> > > >>
> > > >
> > > > --
> > > > Best regards,
> > > > Andrey V. Mashenkov
> > > 
> > >   Конец пересылаемого сообщения 
> > > 
> > > >>>
> > >
> > >
> > >
> > >
> >
>


-- 

Best regards,
Alexei Scherbakov