Re: [DISCUSSION] Add index rebuild time metrics

[ткаленко кирилл]

Hi, Ivan!

What precision would be sufficient?
> If the progress is very slow, I don't see issues with tracking it if the
> percentage float has enough precision.

I think we can add a mention getting cache size.
> 1. Gain an understanding that local cache size
> (CacheMetricsImpl#getCacheSize) should be used as a 100% milestone (it
> isn't mentioned neither in javadoc nor in JMX method description).

Do you think users collect metrics with their hands? I think this is done by 
other systems, such as zabbix.
> 2. Manually calculate sum of all metrics and divide to sum of all cache
> sizes.

As a compromise, I can add jmx methods (rebuilding indexes in the process and 
the percentage of rebuilding) for the entire node, but I tried to find a 
suitable place and did not find it, tell me where to add it?
> On the other hand, % of index rebuild progress is self-descriptive. I don't
> understand why we tend to make user's life harder.

10.08.2020, 21:57, "Ivan Rakov" :
>>  This metric can be used only for local node, to get size of cache use
>>  org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.
>
>  Got it, agree.
>
> If there is a lot of data in node that can be rebuilt, percentage may
>>  change very rarely and may not give an estimate of how much time is left.
>>  If we see for example that 50_000 keys are rebuilt once a minute, and we
>>  have 1_000_000_000 keys, then we can have an approximate estimate. What do
>>  you think of that?
>
> If the progress is very slow, I don't see issues with tracking it if the
> percentage float has enough precision.
> Still, usability of the metric concerns me. In order to estimate remaining
> time of index rebuild, user should:
> 1. Gain an understanding that local cache size
> (CacheMetricsImpl#getCacheSize) should be used as a 100% milestone (it
> isn't mentioned neither in javadoc nor in JMX method description).
> 2. Manually calculate sum of all metrics and divide to sum of all cache
> sizes.
> On the other hand, % of index rebuild progress is self-descriptive. I don't
> understand why we tend to make user's life harder.
>
> --
> Best regards,
> Ivan
>
> On Mon, Aug 10, 2020 at 8:53 PM ткаленко кирилл 
> wrote:
>
>>  Hi, Ivan!
>>
>>  For this you can use
>>  org.apache.ignite.cache.CacheMetrics#IsIndexRebuildInProgress
>>  > How can a local number of processed keys can help us to understand when
>>  > index rebuild will be finished?
>>
>>  This metric can be used only for local node, to get size of cache use
>>  org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.
>>  > We can't compare metric value with cache.size(). First one is node-local,
>>  > while cache size covers all partitions in the cluster.
>>
>>  If there is a lot of data in node that can be rebuilt, percentage may
>>  change very rarely and may not give an estimate of how much time is left.
>>  If we see for example that 50_000 keys are rebuilt once a minute, and we
>>  have 1_000_000_000 keys, then we can have an approximate estimate. What do
>>  you think of that?
>>  > I find one single metric much more usable. It would be perfect if metric
>>  > value is represented in percentage, e.g. current progress of local node
>>  > index rebuild is 60%.
>>
>>  10.08.2020, 19:11, "Ivan Rakov" :
>>  > Folks,
>>  >
>>  > Sorry for coming late to the party. I've taken a look at this issue
>>  during
>>  > review.
>>  >
>>  > How can a local number of processed keys can help us to understand when
>>  > index rebuild will be finished?
>>  > We can't compare metric value with cache.size(). First one is node-local,
>>  > while cache size covers all partitions in the cluster.
>>  > Also, I don't understand why we need to keep separate metrics for all
>>  > caches. Of course, the metric becomes more fair, but obviously harder to
>>  > make conclusions on whether "the index rebuild" process is over (and the
>>  > cluster is ready to process queries quickly).
>>  >
>>  > I find one single metric much more usable. It would be perfect if metric
>>  > value is represented in percentage, e.g. current progress of local node
>>  > index rebuild is 60%.
>>  >
>>  > --
>>  > Best regards,
>>  > Ivan
>>  >
>>  > On Fri, Jul 24, 2020 at 1:35 PM Stanislav Lukyanov <
>>  stanlukya...@gmail.com>
>>  > wrote:
>>  >
>>  >> Got it. I thought that index building and index rebuilding are
>>  essentially
>>  >> the same,
>>  >> but now I see that they are different: index rebuilding cares about all
>>  >> indexes at once while index building cares about particular ones.
>>  >>
>>  >> Kirill's approach sounds good.
>>  >>
>>  >> Stan
>>  >>
>>  >> > On 20 Jul 2020, at 14:54, Alexey Goncharuk <
>>  alexey.goncha...@gmail.com>
>>  >> wrote:
>>  >> >
>>  >> > Stan,
>>  >> >
>>  >> > Currently we never build indexes one-by-one - we always use a cache
>>  data
>>  >> > row visitor which either updates all indexes (see
>>  >> IndexRebuildFullClosure)
>>  >> > or updates a set of all indexes that need 

[jira] [Created] (IGNITE-13346) “Joining persistence node to in-memory cluster couldn't be allowed” Verification is too strict

[YuJue Li (Jira)]

YuJue Li created IGNITE-13346:
-

 Summary: “Joining persistence node to in-memory cluster couldn't 
be allowed” Verification is too strict
 Key: IGNITE-13346
 URL: https://issues.apache.org/jira/browse/IGNITE-13346
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Affects Versions: 2.8.1
Reporter: YuJue Li
 Fix For: 2.10


1.Start a node that turns on persistence and then activate

2.Start visor and connect to cluster

3.Stop the node and restart the node

4.the node start failer and the exception are as folloews:

Caused by: class org.apache.ignite.spi.IgniteSpiException: Joining persistence 
node to in-memory cluster couldn't be allowed due to baseline auto-adjust is 
enabled and timeout equal to 0

I think there may be problems with the verification logic here. There are two 
possible solutions:

1.Relax the restrictions and exclude the client nodes.

2.Point out the IP address corresponding to the in-memory cluster node, which 
is convenient for troubleshooting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[MTCGA]: new failures in builds [5530982] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New test failure in master 
TcpDiscoveryPendingMessageDeliveryTest.testDeliveryAllFailedMessagesInCorrectOrder
 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6369108776551015250=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 06:29:20 11-08-2020 

Re: Apache Ignite 3.0

[Saikat Maitra]

Hi Val,

Thank you for sharing the page and your efforts in compiling the features
list, I am thinking since it is a major version release then shall we
include a section for deprecated features and add changes as mentioned in
our Apache Ignite 3.0 Wishlist

https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist

Regards,
Saikat


On Mon, Aug 10, 2020 at 1:25 AM Petr Ivanov  wrote:

> Hi, Val!
> Thanks for your efforts on this endeavour!
>
>
> I would like to suggest deliveries changes in Apache Ignite 3.0:
>  — modularised  binary delivery — single minimal binary for starting
> Ignite and all other modules and parts of the project (benchmarks,
> examples, etc.) packed in their own binary which can be added via custom
> dependency management tool (i.e. modules.sh)
>  — same distribution for RPM and DEB packages but with modules packed as
> separate ones (PHP for example)
>  — separate thin client release cycle with custom versioning
> Possibly, we can we add additional section to the document you introduced
> for this part.
>
> Also, it seems that full JDK11 support (including building) would be a
> huge milestone and a sign of healthy modern project that tends to be on the
> verge of mainstream technologies and not the stockpile of legacy leftovers
> (fully support Iliya in removing all that was deprecated and/or marked as
> unused anymore).
>
>
> > On 8 Aug 2020, at 02:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >
> > Igniters,
> >
> > I've created the page:
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
> >
> > That's not everything I have in mind, but I believe there is already a
> lot
> > to talk about :)
> >
> > Please take a look let me know if you have any concerns, objections, or
> > questions. Once we reach the consensus on the proposed changes, I will
> > start creating tickets in Jira and a more detailed plan.
> >
> > -Val
> >
> > On Thu, Aug 6, 2020 at 6:28 PM Saikat Maitra 
> > wrote:
> >
> >> Hi Denis, Val
> >>
> >> Thank you for your reply and really appreciate it. It will be very cool
> to
> >> be able to connect and plan release together and learn more about
> Ignite in
> >> the process :)
> >>
> >> Regards
> >> Saikat
> >>
> >>
> >>
> >> On Thu, Aug 6, 2020 at 7:12 PM Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com> wrote:
> >>
> >>> Hi Saikat,
> >>>
> >>> That surely is a great idea. We will work together with Denis on
> setting
> >>> this up in the nearest future.
> >>>
> >>> -Val
> >>>
> >>> On Thu, Aug 6, 2020 at 10:21 AM Denis Magda  wrote:
> >>>
>  Saikat,
> 
>  Fully support your idea on a virtual meetup! Once Val collects and
> >>> outlines
>  the main changes with directions on wiki, we’ll go ahead and schedule
> >> the
>  meetup to talk things out in a bit more detail. We’ll use our new
> >> Virtual
>  Ignite Meetup group for that inviting both Ignite contributors and
>  application developers.
> 
>  Denis
> 
>  On Thursday, August 6, 2020, Saikat Maitra 
>  wrote:
> 
> > Hi Valentin
> >
> > Thank you for sharing and starting the thread. I am thinking if it
> >> will
>  be
> > a good idea to have a virtual meet setup to discuss on the release
> > planning.
> >
> > It will help to learn more individual features to be added and also
> >> to
> > understand about features that have been deprecated and scheduled for
> > removal in Ignite 3.0 release. Also it will help community member to
> > connect in real time and ask questions and share feedback.
> >
> > Regards,
> > Saikat
> >
> > On Thu, Aug 6, 2020 at 3:51 AM Ilya Kasnacheev <
>  ilya.kasnach...@gmail.com>
> > wrote:
> >
> >> Hello!
> >>
> >> I hope to see Apache Ignite release 3.0 as API trimming release.
> >> Let
> >>> us
> >> correct external and internal APIs for which we have better ideas
> >>> now,
>  as
> >> well as remove old and deprecated code.
> >>
> >> We may also introduce new configuration mechanisms and user-facing
> >>> API
> >> (such as cache-less native SQL queries), but this we could
> >> prototype
> > before
> >> starting the 3.0 task.
> >>
> >> I will advise against targeting large new features at 3.0. They can
> >>> be
> >> added in subsequent point releases, whereas we can't really remove
> >> or
> >> remodel stuff in point releases.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> чт, 6 авг. 2020 г. в 03:54, Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com>:
> >>
> >>> Igniters,
> >>>
> >>> I would like to kick off a discussion regarding Ignite 3.0.
> >> Ignite
>  2.0
> >>> exists for more than 3 years now and we've already collected a
> >> significant
> >>> list [1] of changes that we would like to have, but cannot
> >>> implement
> >>> without 

Re: Apache Ignite Board report, August, 2020

[Saikat Maitra]

Hi Dmitriy, Denis

I am thinking if it would be good to add that this year a dedicated track
for Apache Ignite is available in APACHECON 2020 @Home on September 29th.

https://apachecon.com/acah2020/tracks/

Regards,
Saikat

On Mon, Aug 10, 2020 at 7:18 PM Denis Magda  wrote:

> >
> > A little bit off-top: PMCs, would anybody like to be a PMC representative
> > for the new group in addition to Denis M.? It is recommended to have 2
> > members at least to make sure that an event using the Apache-project
> brand
> > follows branding policy and all other Apache guidelines?
>
>
> This should be unnecessary from the branding perspective as long as the
> meetup group description already follows basic trademark guidelines
> (see, a Cassandra
> meetup group  for
> comparison). Based on my experience this is the primary rule to follow. As
> for speakers, they are free to talk about Ignite without any approval by
> the Ignite PMC or ASF Legal. Please let me know if I'm missing anything.
>
> In the future, we might need help with event coordination and organization
> and will be happy to see Ignite committers and PMC members among
> organizers. But as of now, it's premature since the group has just been
> started.
>
> -
> Denis
>
>
> On Mon, Aug 10, 2020 at 2:45 PM Dmitriy Pavlov  wrote:
>
> > Hi Denis,
> >
> > thank you for your valuable feedback, this makes perfect sense to include
> > all these points.
> >
> > A little bit off-top: PMCs, would anybody like to be a PMC representative
> > for the new group in addition to Denis M.? It is recommended to have 2
> > members at least to make sure that an event using the Apache-project
> brand
> > follows branding policy and all other Apache guidelines?
> >
> > Anything else to add to the report?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 10 авг. 2020 г. в 23:27, Denis Magda :
> >
> > > Hi Dmitry,
> > >
> > > Thanks for sending the draft. I would suggest mentioning the following
> in
> > > the project activity section:
> > >
> > >- Ignite Extensions project that decouples the core features from
> > >3rd-party integrations is being moved forward rapidly
> > ><
> https://samaitra.blogspot.com/2020/08/apache-ignite-extensions.html
> > >.
> > >- Community members started the Virtual Apache Ignite Meetup
> > > group that
> is
> > >envisioned as a place to meet and discuss Ignite best practices,
> > > internals,
> > >use cases, share experiences.
> > >- Ignite documentation is being migrated from the readme.io to the
> > >Ignite codebase. The first version is planned to be released with
> > Ignite
> > >2.9 release.
> > >
> > > Also, I would just put the definition of Ignite from the website in the
> > > "description" section. That's what usually PMCs of other projects do in
> > > their reports:
> > >
> > > *Apache Ignite® is a horizontally scalable, fault-tolerant distributed
> > > in-memory computing platform for building real-time applications that
> can
> > > process terabytes of data with in-memory speed.*
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Aug 10, 2020 at 12:14 PM Dmitriy Pavlov 
> > > wrote:
> > >
> > > > Dear Apache Ignite community,
> > > >
> > > > PMC should submit the board report, not later than 12 August.
> > > >
> > > > Previous board with drafted publicly with sharing with all developers
> > > > community. Let us try it one more time. Please share your thoughts on
> > > what
> > > > might be added to the board meeting agenda.
> > > >
> > > > ## Description:
> > > > The mission of Ignite is the creation and maintenance of software
> > related
> > > > to
> > > > High-performance, integrated and distributed in-memory platform for
> > > > computing
> > > > and transacting on large-scale data sets in real-time
> > > >
> > > > ## Issues:
> > > > There are no issues requiring board attention. (hopefully)
> > > >
> > > > ## Membership Data:
> > > > Apache Ignite was founded 2015-08-19 (5 years ago)
> > > > There are currently 53 committers and 34 PMC members in this project.
> > > > The Committer-to-PMC ratio is roughly 7:5.
> > > >
> > > > Community changes, past quarter:
> > > > - No new PMC members. Last addition was Maxim Muzafarov on
> 2020-04-28.
> > > > - Artem Budnikov was added as committer on 2020-07-14
> > > > - Sergey Chugunov was added as committer on 2020-06-12
> > > >
> > > > ## Project Activity:
> > > > Apache Ignite community is working on preparing release 2.9.0.
> > > > Release 3.0 discussion has been resurrected.
> > > > Project documentation migration from readme.io to the code started.
> > > >
> > > > ## Community Health:
> > > > There is a decrease in mailing list traffic comparing to the previous
> > > > quarter (-10-20%). Activity related to code commits remains stable.
> > > > The first committer was elected because of merit related to the
> > non-code
> > > > contribution, but 

Re: IGNITE-12363 Migrate Camel module to ignite-extensions

[Saikat Maitra]

Hi Denis,

Thank you for reviewing the changes.

I will go ahead and merge the changes.

Regards,
Saikat

On Mon, Aug 10, 2020 at 12:14 PM Denis Magda  wrote:

> Saikat,
>
> Thanks for moving Camel to the extensions repo. Please feel free to merge
> the change. Looks good to me.
>
> -
> Denis
>
>
> On Sat, Aug 8, 2020 at 10:38 AM Saikat Maitra 
> wrote:
>
> > Hi,
> >
> > I have created the following PRs for migration of camel module.
> >
> > Jira https://issues.apache.org/jira/browse/IGNITE-12363
> >
> > PRs
> >
> > https://github.com/apache/ignite/pull/8132
> > https://github.com/apache/ignite-extensions/pull/19
> >
> > Please review and share feedback.
> >
> > Regards,
> > Saikat
> >
>

Re: [BLOG] Apache Ignite Extensions

[Saikat Maitra]

Hi Denis,

Thank you so much and glad you shared the article with the user community.

We are close to finishing up the migration of modules as planned and will
start preparing for releases.

Regards,
Saikat


On Mon, Aug 10, 2020 at 12:12 PM Denis Magda  wrote:

> Hi Saikat,
>
> The weekly routines of an Apache Ignite committer :) I believe that Ignite
> application developers would love to learn insights from this article and
> eventually get modular-based Ignite integrations. So, with(out) your
> permission sharing the article via @user list.
>
> Excellent article and the summary of this undergoing project. Keep it up!
>
> -
> Denis
>
>
> On Sat, Aug 8, 2020 at 1:37 PM Saikat Maitra 
> wrote:
>
> > Hi,
> >
> > I have published a blog on Apache Ignite Extensions and about migration
> of
> > existing integration modules to new Ignite Extensions repository.
> >
> > https://samaitra.blogspot.com/2020/08/apache-ignite-extensions.html
> >
> > Please review and share if you have any feedback.
> >
> > Regards,
> > Saikat
> >
>

Re: Apache Ignite Board report, August, 2020

[Denis Magda]

>
> A little bit off-top: PMCs, would anybody like to be a PMC representative
> for the new group in addition to Denis M.? It is recommended to have 2
> members at least to make sure that an event using the Apache-project brand
> follows branding policy and all other Apache guidelines?


This should be unnecessary from the branding perspective as long as the
meetup group description already follows basic trademark guidelines
(see, a Cassandra
meetup group  for
comparison). Based on my experience this is the primary rule to follow. As
for speakers, they are free to talk about Ignite without any approval by
the Ignite PMC or ASF Legal. Please let me know if I'm missing anything.

In the future, we might need help with event coordination and organization
and will be happy to see Ignite committers and PMC members among
organizers. But as of now, it's premature since the group has just been
started.

-
Denis


On Mon, Aug 10, 2020 at 2:45 PM Dmitriy Pavlov  wrote:

> Hi Denis,
>
> thank you for your valuable feedback, this makes perfect sense to include
> all these points.
>
> A little bit off-top: PMCs, would anybody like to be a PMC representative
> for the new group in addition to Denis M.? It is recommended to have 2
> members at least to make sure that an event using the Apache-project brand
> follows branding policy and all other Apache guidelines?
>
> Anything else to add to the report?
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 10 авг. 2020 г. в 23:27, Denis Magda :
>
> > Hi Dmitry,
> >
> > Thanks for sending the draft. I would suggest mentioning the following in
> > the project activity section:
> >
> >- Ignite Extensions project that decouples the core features from
> >3rd-party integrations is being moved forward rapidly
> > >.
> >- Community members started the Virtual Apache Ignite Meetup
> > group that is
> >envisioned as a place to meet and discuss Ignite best practices,
> > internals,
> >use cases, share experiences.
> >- Ignite documentation is being migrated from the readme.io to the
> >Ignite codebase. The first version is planned to be released with
> Ignite
> >2.9 release.
> >
> > Also, I would just put the definition of Ignite from the website in the
> > "description" section. That's what usually PMCs of other projects do in
> > their reports:
> >
> > *Apache Ignite® is a horizontally scalable, fault-tolerant distributed
> > in-memory computing platform for building real-time applications that can
> > process terabytes of data with in-memory speed.*
> >
> > -
> > Denis
> >
> >
> > On Mon, Aug 10, 2020 at 12:14 PM Dmitriy Pavlov 
> > wrote:
> >
> > > Dear Apache Ignite community,
> > >
> > > PMC should submit the board report, not later than 12 August.
> > >
> > > Previous board with drafted publicly with sharing with all developers
> > > community. Let us try it one more time. Please share your thoughts on
> > what
> > > might be added to the board meeting agenda.
> > >
> > > ## Description:
> > > The mission of Ignite is the creation and maintenance of software
> related
> > > to
> > > High-performance, integrated and distributed in-memory platform for
> > > computing
> > > and transacting on large-scale data sets in real-time
> > >
> > > ## Issues:
> > > There are no issues requiring board attention. (hopefully)
> > >
> > > ## Membership Data:
> > > Apache Ignite was founded 2015-08-19 (5 years ago)
> > > There are currently 53 committers and 34 PMC members in this project.
> > > The Committer-to-PMC ratio is roughly 7:5.
> > >
> > > Community changes, past quarter:
> > > - No new PMC members. Last addition was Maxim Muzafarov on 2020-04-28.
> > > - Artem Budnikov was added as committer on 2020-07-14
> > > - Sergey Chugunov was added as committer on 2020-06-12
> > >
> > > ## Project Activity:
> > > Apache Ignite community is working on preparing release 2.9.0.
> > > Release 3.0 discussion has been resurrected.
> > > Project documentation migration from readme.io to the code started.
> > >
> > > ## Community Health:
> > > There is a decrease in mailing list traffic comparing to the previous
> > > quarter (-10-20%). Activity related to code commits remains stable.
> > > The first committer was elected because of merit related to the
> non-code
> > > contribution, but efforts in the documentation.
> > > Apache Ignite users, developers, and PMC members continued to give
> talks
> > > related to the product in C-19 situation mostly online. See also
> > > https://ignite.apache.org/events.html
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> >
>

Re: Apache Ignite Board report, August, 2020

[Dmitriy Pavlov]

Hi Denis,

thank you for your valuable feedback, this makes perfect sense to include
all these points.

A little bit off-top: PMCs, would anybody like to be a PMC representative
for the new group in addition to Denis M.? It is recommended to have 2
members at least to make sure that an event using the Apache-project brand
follows branding policy and all other Apache guidelines?

Anything else to add to the report?

Sincerely,
Dmitriy Pavlov

пн, 10 авг. 2020 г. в 23:27, Denis Magda :

> Hi Dmitry,
>
> Thanks for sending the draft. I would suggest mentioning the following in
> the project activity section:
>
>- Ignite Extensions project that decouples the core features from
>3rd-party integrations is being moved forward rapidly
>.
>- Community members started the Virtual Apache Ignite Meetup
> group that is
>envisioned as a place to meet and discuss Ignite best practices,
> internals,
>use cases, share experiences.
>- Ignite documentation is being migrated from the readme.io to the
>Ignite codebase. The first version is planned to be released with Ignite
>2.9 release.
>
> Also, I would just put the definition of Ignite from the website in the
> "description" section. That's what usually PMCs of other projects do in
> their reports:
>
> *Apache Ignite® is a horizontally scalable, fault-tolerant distributed
> in-memory computing platform for building real-time applications that can
> process terabytes of data with in-memory speed.*
>
> -
> Denis
>
>
> On Mon, Aug 10, 2020 at 12:14 PM Dmitriy Pavlov 
> wrote:
>
> > Dear Apache Ignite community,
> >
> > PMC should submit the board report, not later than 12 August.
> >
> > Previous board with drafted publicly with sharing with all developers
> > community. Let us try it one more time. Please share your thoughts on
> what
> > might be added to the board meeting agenda.
> >
> > ## Description:
> > The mission of Ignite is the creation and maintenance of software related
> > to
> > High-performance, integrated and distributed in-memory platform for
> > computing
> > and transacting on large-scale data sets in real-time
> >
> > ## Issues:
> > There are no issues requiring board attention. (hopefully)
> >
> > ## Membership Data:
> > Apache Ignite was founded 2015-08-19 (5 years ago)
> > There are currently 53 committers and 34 PMC members in this project.
> > The Committer-to-PMC ratio is roughly 7:5.
> >
> > Community changes, past quarter:
> > - No new PMC members. Last addition was Maxim Muzafarov on 2020-04-28.
> > - Artem Budnikov was added as committer on 2020-07-14
> > - Sergey Chugunov was added as committer on 2020-06-12
> >
> > ## Project Activity:
> > Apache Ignite community is working on preparing release 2.9.0.
> > Release 3.0 discussion has been resurrected.
> > Project documentation migration from readme.io to the code started.
> >
> > ## Community Health:
> > There is a decrease in mailing list traffic comparing to the previous
> > quarter (-10-20%). Activity related to code commits remains stable.
> > The first committer was elected because of merit related to the non-code
> > contribution, but efforts in the documentation.
> > Apache Ignite users, developers, and PMC members continued to give talks
> > related to the product in C-19 situation mostly online. See also
> > https://ignite.apache.org/events.html
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
>

[MTCGA]: new failures in builds [5530437] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New test failure in master 
TcpDiscoveryPendingMessageDeliveryTest.testPendingMessagesOverflow 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=528088480146099042=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 23:44:19 10-08-2020 

Re: Apache Ignite Board report, August, 2020

[Denis Magda]

Hi Dmitry,

Thanks for sending the draft. I would suggest mentioning the following in
the project activity section:

   - Ignite Extensions project that decouples the core features from
   3rd-party integrations is being moved forward rapidly
   .
   - Community members started the Virtual Apache Ignite Meetup
    group that is
   envisioned as a place to meet and discuss Ignite best practices, internals,
   use cases, share experiences.
   - Ignite documentation is being migrated from the readme.io to the
   Ignite codebase. The first version is planned to be released with Ignite
   2.9 release.

Also, I would just put the definition of Ignite from the website in the
"description" section. That's what usually PMCs of other projects do in
their reports:

*Apache Ignite® is a horizontally scalable, fault-tolerant distributed
in-memory computing platform for building real-time applications that can
process terabytes of data with in-memory speed.*

-
Denis


On Mon, Aug 10, 2020 at 12:14 PM Dmitriy Pavlov  wrote:

> Dear Apache Ignite community,
>
> PMC should submit the board report, not later than 12 August.
>
> Previous board with drafted publicly with sharing with all developers
> community. Let us try it one more time. Please share your thoughts on what
> might be added to the board meeting agenda.
>
> ## Description:
> The mission of Ignite is the creation and maintenance of software related
> to
> High-performance, integrated and distributed in-memory platform for
> computing
> and transacting on large-scale data sets in real-time
>
> ## Issues:
> There are no issues requiring board attention. (hopefully)
>
> ## Membership Data:
> Apache Ignite was founded 2015-08-19 (5 years ago)
> There are currently 53 committers and 34 PMC members in this project.
> The Committer-to-PMC ratio is roughly 7:5.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Maxim Muzafarov on 2020-04-28.
> - Artem Budnikov was added as committer on 2020-07-14
> - Sergey Chugunov was added as committer on 2020-06-12
>
> ## Project Activity:
> Apache Ignite community is working on preparing release 2.9.0.
> Release 3.0 discussion has been resurrected.
> Project documentation migration from readme.io to the code started.
>
> ## Community Health:
> There is a decrease in mailing list traffic comparing to the previous
> quarter (-10-20%). Activity related to code commits remains stable.
> The first committer was elected because of merit related to the non-code
> contribution, but efforts in the documentation.
> Apache Ignite users, developers, and PMC members continued to give talks
> related to the product in C-19 situation mostly online. See also
> https://ignite.apache.org/events.html
>
> Sincerely,
> Dmitriy Pavlov
>

Apache Ignite Board report, August, 2020

[Dmitriy Pavlov]

Dear Apache Ignite community,

PMC should submit the board report, not later than 12 August.

Previous board with drafted publicly with sharing with all developers
community. Let us try it one more time. Please share your thoughts on what
might be added to the board meeting agenda.

## Description:
The mission of Ignite is the creation and maintenance of software related
to
High-performance, integrated and distributed in-memory platform for
computing
and transacting on large-scale data sets in real-time

## Issues:
There are no issues requiring board attention. (hopefully)

## Membership Data:
Apache Ignite was founded 2015-08-19 (5 years ago)
There are currently 53 committers and 34 PMC members in this project.
The Committer-to-PMC ratio is roughly 7:5.

Community changes, past quarter:
- No new PMC members. Last addition was Maxim Muzafarov on 2020-04-28.
- Artem Budnikov was added as committer on 2020-07-14
- Sergey Chugunov was added as committer on 2020-06-12

## Project Activity:
Apache Ignite community is working on preparing release 2.9.0.
Release 3.0 discussion has been resurrected.
Project documentation migration from readme.io to the code started.

## Community Health:
There is a decrease in mailing list traffic comparing to the previous
quarter (-10-20%). Activity related to code commits remains stable.
The first committer was elected because of merit related to the non-code
contribution, but efforts in the documentation.
Apache Ignite users, developers, and PMC members continued to give talks
related to the product in C-19 situation mostly online. See also
https://ignite.apache.org/events.html

Sincerely,
Dmitriy Pavlov

Re: [DISCUSSION] Add index rebuild time metrics

[Ivan Rakov]

>
> This metric can be used only for local node, to get size of cache use
> org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.

 Got it, agree.

If there is a lot of data in node that can be rebuilt, percentage may
> change very rarely and may not give an estimate of how much time is left.
> If we see for example that 50_000 keys are rebuilt once a minute, and we
> have 1_000_000_000 keys, then we can have an approximate estimate. What do
> you think of that?

If the progress is very slow, I don't see issues with tracking it if the
percentage float has enough precision.
Still, usability of the metric concerns me. In order to estimate remaining
time of index rebuild, user should:
1. Gain an understanding that local cache size
(CacheMetricsImpl#getCacheSize) should be used as a 100% milestone (it
isn't mentioned neither in javadoc nor in JMX method description).
2. Manually calculate sum of all metrics and divide to sum of all cache
sizes.
On the other hand, % of index rebuild progress is self-descriptive. I don't
understand why we tend to make user's life harder.

--
Best regards,
Ivan


On Mon, Aug 10, 2020 at 8:53 PM ткаленко кирилл 
wrote:

> Hi, Ivan!
>
> For this you can use
> org.apache.ignite.cache.CacheMetrics#IsIndexRebuildInProgress
> > How can a local number of processed keys can help us to understand when
> > index rebuild will be finished?
>
> This metric can be used only for local node, to get size of cache use
> org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.
> > We can't compare metric value with cache.size(). First one is node-local,
> > while cache size covers all partitions in the cluster.
>
> If there is a lot of data in node that can be rebuilt, percentage may
> change very rarely and may not give an estimate of how much time is left.
> If we see for example that 50_000 keys are rebuilt once a minute, and we
> have 1_000_000_000 keys, then we can have an approximate estimate. What do
> you think of that?
> > I find one single metric much more usable. It would be perfect if metric
> > value is represented in percentage, e.g. current progress of local node
> > index rebuild is 60%.
>
> 10.08.2020, 19:11, "Ivan Rakov" :
> > Folks,
> >
> > Sorry for coming late to the party. I've taken a look at this issue
> during
> > review.
> >
> > How can a local number of processed keys can help us to understand when
> > index rebuild will be finished?
> > We can't compare metric value with cache.size(). First one is node-local,
> > while cache size covers all partitions in the cluster.
> > Also, I don't understand why we need to keep separate metrics for all
> > caches. Of course, the metric becomes more fair, but obviously harder to
> > make conclusions on whether "the index rebuild" process is over (and the
> > cluster is ready to process queries quickly).
> >
> > I find one single metric much more usable. It would be perfect if metric
> > value is represented in percentage, e.g. current progress of local node
> > index rebuild is 60%.
> >
> > --
> > Best regards,
> > Ivan
> >
> > On Fri, Jul 24, 2020 at 1:35 PM Stanislav Lukyanov <
> stanlukya...@gmail.com>
> > wrote:
> >
> >>  Got it. I thought that index building and index rebuilding are
> essentially
> >>  the same,
> >>  but now I see that they are different: index rebuilding cares about all
> >>  indexes at once while index building cares about particular ones.
> >>
> >>  Kirill's approach sounds good.
> >>
> >>  Stan
> >>
> >>  > On 20 Jul 2020, at 14:54, Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> >>  wrote:
> >>  >
> >>  > Stan,
> >>  >
> >>  > Currently we never build indexes one-by-one - we always use a cache
> data
> >>  > row visitor which either updates all indexes (see
> >>  IndexRebuildFullClosure)
> >>  > or updates a set of all indexes that need to catch up (see
> >>  > IndexRebuildPartialClosure). GIven that, I do not see any need for
> >>  > per-index rebuild status as this status will be updated for all
> outdated
> >>  > indexes simultaneously.
> >>  >
> >>  > Kirill's approach for the total number of processed keys per cache
> seems
> >>  > reasonable to me.
> >>  >
> >>  > --AG
> >>  >
> >>  > пт, 3 июл. 2020 г. в 10:12, ткаленко кирилл :
> >>  >
> >>  >> Hi, Stan!
> >>  >>
> >>  >> Perhaps it is worth clarifying what exactly I wanted to say.
> >>  >> Now we have 2 processes: building and rebuilding indexes.
> >>  >>
> >>  >> At moment, we have some metrics for rebuilding indexes:
> >>  >> "IsIndexRebuildInProgress", "IndexBuildCountPartitionsLeft".
> >>  >>
> >>  >> I suggest adding another metric "Indexrebuildkeyprocessed", which
> will
> >>  >> allow you to determine how many records are left to rebuild for
> cache.
> >>  >>
> >>  >> I think your comments are more about building an index that may need
> >>  more
> >>  >> metrics, but I think you should do it in a separate ticket.
> >>  >>
> >>  >> 03.07.2020, 03:09, "Stanislav Lukyanov" :
> >>  >>> If multiple 

Re: [DISCUSSION] Add index rebuild time metrics

[ткаленко кирилл]

Hi, Ivan!

For this you can use 
org.apache.ignite.cache.CacheMetrics#IsIndexRebuildInProgress
> How can a local number of processed keys can help us to understand when
> index rebuild will be finished?

This metric can be used only for local node, to get size of cache use 
org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.
> We can't compare metric value with cache.size(). First one is node-local,
> while cache size covers all partitions in the cluster.

If there is a lot of data in node that can be rebuilt, percentage may change 
very rarely and may not give an estimate of how much time is left. If we see 
for example that 50_000 keys are rebuilt once a minute, and we have 
1_000_000_000 keys, then we can have an approximate estimate. What do you think 
of that?
> I find one single metric much more usable. It would be perfect if metric
> value is represented in percentage, e.g. current progress of local node
> index rebuild is 60%.

10.08.2020, 19:11, "Ivan Rakov" :
> Folks,
>
> Sorry for coming late to the party. I've taken a look at this issue during
> review.
>
> How can a local number of processed keys can help us to understand when
> index rebuild will be finished?
> We can't compare metric value with cache.size(). First one is node-local,
> while cache size covers all partitions in the cluster.
> Also, I don't understand why we need to keep separate metrics for all
> caches. Of course, the metric becomes more fair, but obviously harder to
> make conclusions on whether "the index rebuild" process is over (and the
> cluster is ready to process queries quickly).
>
> I find one single metric much more usable. It would be perfect if metric
> value is represented in percentage, e.g. current progress of local node
> index rebuild is 60%.
>
> --
> Best regards,
> Ivan
>
> On Fri, Jul 24, 2020 at 1:35 PM Stanislav Lukyanov 
> wrote:
>
>>  Got it. I thought that index building and index rebuilding are essentially
>>  the same,
>>  but now I see that they are different: index rebuilding cares about all
>>  indexes at once while index building cares about particular ones.
>>
>>  Kirill's approach sounds good.
>>
>>  Stan
>>
>>  > On 20 Jul 2020, at 14:54, Alexey Goncharuk 
>>  wrote:
>>  >
>>  > Stan,
>>  >
>>  > Currently we never build indexes one-by-one - we always use a cache data
>>  > row visitor which either updates all indexes (see
>>  IndexRebuildFullClosure)
>>  > or updates a set of all indexes that need to catch up (see
>>  > IndexRebuildPartialClosure). GIven that, I do not see any need for
>>  > per-index rebuild status as this status will be updated for all outdated
>>  > indexes simultaneously.
>>  >
>>  > Kirill's approach for the total number of processed keys per cache seems
>>  > reasonable to me.
>>  >
>>  > --AG
>>  >
>>  > пт, 3 июл. 2020 г. в 10:12, ткаленко кирилл :
>>  >
>>  >> Hi, Stan!
>>  >>
>>  >> Perhaps it is worth clarifying what exactly I wanted to say.
>>  >> Now we have 2 processes: building and rebuilding indexes.
>>  >>
>>  >> At moment, we have some metrics for rebuilding indexes:
>>  >> "IsIndexRebuildInProgress", "IndexBuildCountPartitionsLeft".
>>  >>
>>  >> I suggest adding another metric "Indexrebuildkeyprocessed", which will
>>  >> allow you to determine how many records are left to rebuild for cache.
>>  >>
>>  >> I think your comments are more about building an index that may need
>>  more
>>  >> metrics, but I think you should do it in a separate ticket.
>>  >>
>>  >> 03.07.2020, 03:09, "Stanislav Lukyanov" :
>>  >>> If multiple indexes are to be built "number of indexed keys" metric may
>>  >> be misleading.
>>  >>>
>>  >>> As a cluster admin, I'd like to know:
>>  >>> - Are all indexes ready on a node?
>>  >>> - How many indexes are to be built?
>>  >>> - How much resources are used by the index building (how many threads
>>  >> are used)?
>>  >>> - Which index(es?) is being built right now?
>>  >>> - How much time until the current (single) index building finishes?
>>  Here
>>  >> "time" can be a lot of things: partitions, entries, percent of the
>>  cache,
>>  >> minutes and hours
>>  >>> - How much time until all indexes are built?
>>  >>> - How much does it take to build each of my indexes / a single index of
>>  >> my cache on average?
>>  >>>
>>  >>> I think we need a set of metrics and/or log messages to solve all of
>>  >> these questions.
>>  >>> I imaging something like:
>>  >>> - numberOfIndexesToBuild
>>  >>> - a standard set of metrics on the index building thread pool (do we
>>  >> already have it?)
>>  >>> - currentlyBuiltIndexName (assuming we only build one at a time which
>>  is
>>  >> probably not true)
>>  >>> - for the "time" metrics I think percentage might be the best as it's
>>  >> the easiest to understand; we may add multiple metrics though.
>>  >>> - For "time per each index" I'd add detailed log messages stating how
>>  >> long did it take to build a particular index
>>  >>>
>>  >>> Thanks,
>>  >>> Stan
>>  

Re: IGNITE-12363 Migrate Camel module to ignite-extensions

[Denis Magda]

Saikat,

Thanks for moving Camel to the extensions repo. Please feel free to merge
the change. Looks good to me.

-
Denis


On Sat, Aug 8, 2020 at 10:38 AM Saikat Maitra 
wrote:

> Hi,
>
> I have created the following PRs for migration of camel module.
>
> Jira https://issues.apache.org/jira/browse/IGNITE-12363
>
> PRs
>
> https://github.com/apache/ignite/pull/8132
> https://github.com/apache/ignite-extensions/pull/19
>
> Please review and share feedback.
>
> Regards,
> Saikat
>

Re: [BLOG] Apache Ignite Extensions

[Denis Magda]

Hi Saikat,

The weekly routines of an Apache Ignite committer :) I believe that Ignite
application developers would love to learn insights from this article and
eventually get modular-based Ignite integrations. So, with(out) your
permission sharing the article via @user list.

Excellent article and the summary of this undergoing project. Keep it up!

-
Denis


On Sat, Aug 8, 2020 at 1:37 PM Saikat Maitra 
wrote:

> Hi,
>
> I have published a blog on Apache Ignite Extensions and about migration of
> existing integration modules to new Ignite Extensions repository.
>
> https://samaitra.blogspot.com/2020/08/apache-ignite-extensions.html
>
> Please review and share if you have any feedback.
>
> Regards,
> Saikat
>

Re: [DISCUSSION] Add index rebuild time metrics

[Ivan Rakov]

Folks,

Sorry for coming late to the party. I've taken a look at this issue during
review.

How can a local number of processed keys can help us to understand when
index rebuild will be finished?
We can't compare metric value with cache.size(). First one is node-local,
while cache size covers all partitions in the cluster.
Also, I don't understand why we need to keep separate metrics for all
caches. Of course, the metric becomes more fair, but obviously harder to
make conclusions on whether "the index rebuild" process is over (and the
cluster is ready to process queries quickly).

I find one single metric much more usable. It would be perfect if metric
value is represented in percentage, e.g. current progress of local node
index rebuild is 60%.

--
Best regards,
Ivan

On Fri, Jul 24, 2020 at 1:35 PM Stanislav Lukyanov 
wrote:

> Got it. I thought that index building and index rebuilding are essentially
> the same,
> but now I see that they are different: index rebuilding cares about all
> indexes at once while index building cares about particular ones.
>
> Kirill's approach sounds good.
>
> Stan
>
> > On 20 Jul 2020, at 14:54, Alexey Goncharuk 
> wrote:
> >
> > Stan,
> >
> > Currently we never build indexes one-by-one - we always use a cache data
> > row visitor which either updates all indexes (see
> IndexRebuildFullClosure)
> > or updates a set of all indexes that need to catch up (see
> > IndexRebuildPartialClosure). GIven that, I do not see any need for
> > per-index rebuild status as this status will be updated for all outdated
> > indexes simultaneously.
> >
> > Kirill's approach for the total number of processed keys per cache seems
> > reasonable to me.
> >
> > --AG
> >
> > пт, 3 июл. 2020 г. в 10:12, ткаленко кирилл :
> >
> >> Hi, Stan!
> >>
> >> Perhaps it is worth clarifying what exactly I wanted to say.
> >> Now we have 2 processes: building and rebuilding indexes.
> >>
> >> At moment, we have some metrics for rebuilding indexes:
> >> "IsIndexRebuildInProgress", "IndexBuildCountPartitionsLeft".
> >>
> >> I suggest adding another metric "Indexrebuildkeyprocessed", which will
> >> allow you to determine how many records are left to rebuild for cache.
> >>
> >> I think your comments are more about building an index that may need
> more
> >> metrics, but I think you should do it in a separate ticket.
> >>
> >> 03.07.2020, 03:09, "Stanislav Lukyanov" :
> >>> If multiple indexes are to be built "number of indexed keys" metric may
> >> be misleading.
> >>>
> >>> As a cluster admin, I'd like to know:
> >>> - Are all indexes ready on a node?
> >>> - How many indexes are to be built?
> >>> - How much resources are used by the index building (how many threads
> >> are used)?
> >>> - Which index(es?) is being built right now?
> >>> - How much time until the current (single) index building finishes?
> Here
> >> "time" can be a lot of things: partitions, entries, percent of the
> cache,
> >> minutes and hours
> >>> - How much time until all indexes are built?
> >>> - How much does it take to build each of my indexes / a single index of
> >> my cache on average?
> >>>
> >>> I think we need a set of metrics and/or log messages to solve all of
> >> these questions.
> >>> I imaging something like:
> >>> - numberOfIndexesToBuild
> >>> - a standard set of metrics on the index building thread pool (do we
> >> already have it?)
> >>> - currentlyBuiltIndexName (assuming we only build one at a time which
> is
> >> probably not true)
> >>> - for the "time" metrics I think percentage might be the best as it's
> >> the easiest to understand; we may add multiple metrics though.
> >>> - For "time per each index" I'd add detailed log messages stating how
> >> long did it take to build a particular index
> >>>
> >>> Thanks,
> >>> Stan
> >>>
>  On 26 Jun 2020, at 12:49, ткаленко кирилл 
> >> wrote:
> 
>  Hi, Igniters.
> 
>  I would like to know if it is possible to estimate how much the index
> >> rebuild will take?
> 
>  At the moment, I have found the following metrics [1] and [2] and
> >> since the rebuild is based on caches, I think it would be useful to know
> >> how many records are processed in indexing. This way we can estimate how
> >> long we have to wait for the index to be rebuilt by subtracting [3] and
> how
> >> many records are indexed.
> 
>  I think we should add this metric [4].
> 
>  Comments, suggestions?
> 
>  [1] - https://issues.apache.org/jira/browse/IGNITE-12184
>  [2] -
> >>
> org.apache.ignite.internal.processors.cache.CacheGroupMetricsImpl#idxBuildCntPartitionsLeft
>  [3] - org.apache.ignite.cache.CacheMetrics#getCacheSize
>  [4] - org.apache.ignite.cache.CacheMetrics#getNumberIndexedKeys
> >>
>
>

Re: Too many messages in log in case of exceptions in user computations.

[Nikolay Izhikov]

Hello, Vasiliy.

This messages are shown on the different nodes, isn’t it?

> 10 авг. 2020 г., в 18:34, Vasiliy Sisko  написал(а):
> 
> In case of errors in user computations the Ignite Compute Grid produces a
> lot of errors into the log.
> In the worst way it produces the next messages:
>1. Failed to execute job: …
>2. Failed to reduce job results: …
>3. Failed to execute task: …
> 
> There is a suggestion to decrease log level for first and second messages
> to DEBUG level, because of the third message will still be shown.
> 
> -- 
> Vasiliy Sisko

Too many messages in log in case of exceptions in user computations.

[Vasiliy Sisko]

 In case of errors in user computations the Ignite Compute Grid produces a
lot of errors into the log.
In the worst way it produces the next messages:
1. Failed to execute job: …
2. Failed to reduce job results: …
3. Failed to execute task: …

There is a suggestion to decrease log level for first and second messages
to DEBUG level, because of the third message will still be shown.

-- 
Vasiliy Sisko

Re: [DISCUSSION] Cache warmup

[ткаленко кирилл]

Great! Then I proceed to ticket 
https://issues.apache.org/jira/browse/IGNITE-13345.

10.08.2020, 16:30, "Stanislav Lukyanov" :
> All of this looks awesome, covers the use cases I know about.
> Thanks!
>
> Stan
>
>>  On 10 Aug 2020, at 15:39, ткаленко кирилл  wrote:
>>
>>  Hi, Stan again :-)
>>
>>  I suggest adding a little more flexibility to configuration:
>>  1)Add default warm-up configuration for all regions into 
>> org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
>>  2)Add a NoOp strategy for turning off heating of a specific region
>>
>>  Thus, when starting warm-up, region configuration is taken at beginning, 
>> and if it is not present, it is taken from default. And if we don't want to 
>> warm up region, we set NoOp.
>>
>>  10.08.2020, 10:20, "ткаленко кирилл" :
>>>  Hi, Stan!
>>>
>>>  As a result of personal correspondence I realized that you are right about 
>>> making changes:
>>>  1)Move warm-up configuration to 
>>> org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>>>  2)Start warming up for each region sequentially;
>>>  3)Improving warm-up interface:
>>>
>>>  package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>>  import org.apache.ignite.IgniteCheckedException;
>>>  import org.apache.ignite.configuration.WarmUpConfiguration;
>>>  import org.apache.ignite.internal.GridKernalContext;
>>>  import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>>>
>>>  /**
>>>   * Interface for warming up.
>>>   */
>>>  public interface WarmUpStrategy {
>>>  /**
>>>   * Returns configuration class for mapping to strategy.
>>>   *
>>>   * @return Configuration class.
>>>   */
>>>  Class configClass();
>>>
>>>  /**
>>>   * Warm up.
>>>   *
>>>   * @param kernalCtx Kernal context.
>>>   * @param cfg Warm-up configuration.
>>>   * @param region Data region.
>>>   * @throws IgniteCheckedException if faild.
>>>   */
>>>  void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) 
>>> throws IgniteCheckedException;
>>>
>>>  /**
>>>   * Closing warm up.
>>>   *
>>>   * @throws IgniteCheckedException if faild.
>>>   */
>>>  void close() throws IgniteCheckedException;
>>>  }
>>>
>>>  4)Add a command to "control.sh", to stop current warm-up and cancel all 
>>> others: --warm-up stop
>>>  5)The "load all" strategy will work as long as there is enough RAM and 
>>> index pages will also take priority.
>>>
>>>  04.08.2020, 13:29, "ткаленко кирилл" :
   Hi, Slava!

   Thank you for looking at the offer and making fair comments.

   I personally discussed with Anton and Alexey because they are author and 
 sponsor of "IEP-40" and we found out that point 2 in it is no longer 
 relevant and it can be removed.
   I suggest implementing point 3, since it may be independent of point 1. 
 Also, the warm-up will always start after restore phase, without 
 subscribing to events.

   You are right this should be mentioned in the documentation and javadoc.
>    This means that the user's thread, which starts Ignite via
>    Ignition.start(), will wait for ana additional step - cache warm-up.
>    I think this fact has to be clearly mentioned in our documentation (at
>    Javadocat least) because this step can be time-consuming.

   My suggestion for implementation:
   1)Adding a marker interface 
 "org.apache.ignite.configuration.WarmUpConfiguration" for configuring 
 cache warming;
   2)Set only one configuration via 
 "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
   3)Add an internal warm-up interface that will start in [1] after [2];

   package org.apache.ignite.internal.processors.cache.warmup;

   import org.apache.ignite.IgniteCheckedException;
   import org.apache.ignite.configuration.WarmUpConfiguration;
   import org.apache.ignite.internal.GridKernalContext;

   /**
    * Interface for warming up.
    */
   public interface WarmUpStrategy {
   /**
    * Returns configuration class for mapping to strategy.
    *
    * @return Configuration class.
    */
   Class configClass();

   /**
    * Warm up.
    *
    * @param kernalCtx Kernal context.
    * @param cfg Warm-up configuration.
    * @throws IgniteCheckedException if faild.
    */
   void warmUp(GridKernalContext kernalCtx, T cfg) throws 
 IgniteCheckedException;
   }

   4)Adding an internal plugin extension for add own strategies;

   package org.apache.ignite.internal.processors.cache.warmup;

   import java.util.Collection;
   import org.apache.ignite.plugin.Extension;

   /**
    * Interface for getting warm-up strategies from plugins.
    

[jira] [Created] (IGNITE-13345) Warming up node

[Kirill Tkalenko (Jira)]

Kirill Tkalenko created IGNITE-13345:


 Summary: Warming up node
 Key: IGNITE-13345
 URL: https://issues.apache.org/jira/browse/IGNITE-13345
 Project: Ignite
  Issue Type: New Feature
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.10


Summary of 
[http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Cache-warmup-td48582.html|Dev-list]:

# Adding a marker interface 
*org.apache.ignite.configuration.WarmUpConfiguratio*n;

# Adding a configuration to
* 
*org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration*
* 
*org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration*

# Add an internal warm-up interface that will start in [1] after [2] (after 
recovery);
{code:java}
package org.apache.ignite.internal.processors.cache.warmup;

import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
import org.apache.ignite.internal.processors.cache.persistence.DataRegion;

/**
 * Interface for warming up.
 */
public interface WarmUpStrategy {
/**
 * Returns configuration class for mapping to strategy.
 *
 * @return Configuration class.
 */
Class configClass();

/**
 * Warm up.
 *
 * @param kernalCtx Kernal context.
 * @param cfg   Warm-up configuration.
 * @param regionData region.
 * @throws IgniteCheckedException if faild.
 */
void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws 
IgniteCheckedException;

/**
 * Closing warm up.
 *
 * @throws IgniteCheckedException if faild.
 */
void close() throws IgniteCheckedException;
}
{code}


# Adding an internal plugin extension for add own strategies;
 

{code:java}
package org.apache.ignite.internal.processors.cache.warmup;
 
import java.util.Collection;
import org.apache.ignite.plugin.Extension;
 
/**
 * Interface for getting warm-up strategies from plugins.
 */
public interface WarmUpStrategySupplier extends Extension {
/**
 * Getting warm-up strategies.
 *
 * @return Warm-up strategies.
 */
Collection strategies();
}
{code}

# Adding strategies:
* Without implementation, for the possibility of disabling the warm-up: 
*org.apache.ignite.internal.processors.cache.warmup.NoOpWarmUp*, 
*org.apache.ignite.internal.processors.cache.warmup.NoOpWarmUpConfiguration*
* Loading everything while there is RAM with priority to indexes: 
*org.apache.ignite.internal.processors.cache.warmup.LoadAllWarmUp*, 
*org.apache.ignite.internal.processors.cache.warmup.LoadAllWarmUpConfiguration*

# Add a command to "control.sh", to stop current warm-up and cancel all others: 
--warm-up stop

[1] - 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
[2] - 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Cache warmup

[Stanislav Lukyanov]

All of this looks awesome, covers the use cases I know about.
Thanks!

Stan

> On 10 Aug 2020, at 15:39, ткаленко кирилл  wrote:
> 
> Hi, Stan again :-)
> 
> I suggest adding a little more flexibility to configuration:
> 1)Add default warm-up configuration for all regions into 
> org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
> 2)Add a NoOp strategy for turning off heating of a specific region
> 
> Thus, when starting warm-up, region configuration is taken at beginning, and 
> if it is not present, it is taken from default. And if we don't want to warm 
> up region, we set NoOp.
> 
> 10.08.2020, 10:20, "ткаленко кирилл" :
>> Hi, Stan!
>> 
>> As a result of personal correspondence I realized that you are right about 
>> making changes:
>> 1)Move warm-up configuration to 
>> org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>> 2)Start warming up for each region sequentially;
>> 3)Improving warm-up interface:
>> 
>> package org.apache.ignite.internal.processors.cache.warmup;
>> 
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.configuration.WarmUpConfiguration;
>> import org.apache.ignite.internal.GridKernalContext;
>> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>> 
>> /**
>>  * Interface for warming up.
>>  */
>> public interface WarmUpStrategy {
>> /**
>>  * Returns configuration class for mapping to strategy.
>>  *
>>  * @return Configuration class.
>>  */
>> Class configClass();
>> 
>> /**
>>  * Warm up.
>>  *
>>  * @param kernalCtx Kernal context.
>>  * @param cfg Warm-up configuration.
>>  * @param region Data region.
>>  * @throws IgniteCheckedException if faild.
>>  */
>> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) 
>> throws IgniteCheckedException;
>> 
>> /**
>>  * Closing warm up.
>>  *
>>  * @throws IgniteCheckedException if faild.
>>  */
>> void close() throws IgniteCheckedException;
>> }
>> 
>> 4)Add a command to "control.sh", to stop current warm-up and cancel all 
>> others: --warm-up stop
>> 5)The "load all" strategy will work as long as there is enough RAM and index 
>> pages will also take priority.
>> 
>> 04.08.2020, 13:29, "ткаленко кирилл" :
>>>  Hi, Slava!
>>> 
>>>  Thank you for looking at the offer and making fair comments.
>>> 
>>>  I personally discussed with Anton and Alexey because they are author and 
>>> sponsor of "IEP-40" and we found out that point 2 in it is no longer 
>>> relevant and it can be removed.
>>>  I suggest implementing point 3, since it may be independent of point 1. 
>>> Also, the warm-up will always start after restore phase, without 
>>> subscribing to events.
>>> 
>>>  You are right this should be mentioned in the documentation and javadoc.
   This means that the user's thread, which starts Ignite via
   Ignition.start(), will wait for ana additional step - cache warm-up.
   I think this fact has to be clearly mentioned in our documentation (at
   Javadocat least) because this step can be time-consuming.
>>> 
>>>  My suggestion for implementation:
>>>  1)Adding a marker interface 
>>> "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache 
>>> warming;
>>>  2)Set only one configuration via 
>>> "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>>  3)Add an internal warm-up interface that will start in [1] after [2];
>>> 
>>>  package org.apache.ignite.internal.processors.cache.warmup;
>>> 
>>>  import org.apache.ignite.IgniteCheckedException;
>>>  import org.apache.ignite.configuration.WarmUpConfiguration;
>>>  import org.apache.ignite.internal.GridKernalContext;
>>> 
>>>  /**
>>>   * Interface for warming up.
>>>   */
>>>  public interface WarmUpStrategy {
>>>  /**
>>>   * Returns configuration class for mapping to strategy.
>>>   *
>>>   * @return Configuration class.
>>>   */
>>>  Class configClass();
>>> 
>>>  /**
>>>   * Warm up.
>>>   *
>>>   * @param kernalCtx Kernal context.
>>>   * @param cfg Warm-up configuration.
>>>   * @throws IgniteCheckedException if faild.
>>>   */
>>>  void warmUp(GridKernalContext kernalCtx, T cfg) throws 
>>> IgniteCheckedException;
>>>  }
>>> 
>>>  4)Adding an internal plugin extension for add own strategies;
>>> 
>>>  package org.apache.ignite.internal.processors.cache.warmup;
>>> 
>>>  import java.util.Collection;
>>>  import org.apache.ignite.plugin.Extension;
>>> 
>>>  /**
>>>   * Interface for getting warm-up strategies from plugins.
>>>   */
>>>  public interface WarmUpStrategySupplier extends Extension {
>>>  /**
>>>   * Getting warm-up strategies.
>>>   *
>>>   * @return Warm-up strategies.
>>>   */
>>>  Collection strategies();
>>>  }
>>> 
>>>  5)Add a "Load all" strategy that will load everything to memory as long as 
>>> there is 

Re: [DISCUSSION] Cache warmup

[ткаленко кирилл]

Hi, Stan again :-)

I suggest adding a little more flexibility to configuration:
1)Add default warm-up configuration for all regions into 
org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
2)Add a NoOp strategy for turning off heating of a specific region

Thus, when starting warm-up, region configuration is taken at beginning, and if 
it is not present, it is taken from default. And if we don't want to warm up 
region, we set NoOp.

10.08.2020, 10:20, "ткаленко кирилл" :
> Hi, Stan!
>
> As a result of personal correspondence I realized that you are right about 
> making changes:
> 1)Move warm-up configuration to 
> org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
> 2)Start warming up for each region sequentially;
> 3)Improving warm-up interface:
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.configuration.WarmUpConfiguration;
> import org.apache.ignite.internal.GridKernalContext;
> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>
> /**
>  * Interface for warming up.
>  */
> public interface WarmUpStrategy {
> /**
>  * Returns configuration class for mapping to strategy.
>  *
>  * @return Configuration class.
>  */
> Class configClass();
>
> /**
>  * Warm up.
>  *
>  * @param kernalCtx Kernal context.
>  * @param cfg Warm-up configuration.
>  * @param region Data region.
>  * @throws IgniteCheckedException if faild.
>  */
> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws 
> IgniteCheckedException;
>
> /**
>  * Closing warm up.
>  *
>  * @throws IgniteCheckedException if faild.
>  */
> void close() throws IgniteCheckedException;
> }
>
> 4)Add a command to "control.sh", to stop current warm-up and cancel all 
> others: --warm-up stop
> 5)The "load all" strategy will work as long as there is enough RAM and index 
> pages will also take priority.
>
> 04.08.2020, 13:29, "ткаленко кирилл" :
>>  Hi, Slava!
>>
>>  Thank you for looking at the offer and making fair comments.
>>
>>  I personally discussed with Anton and Alexey because they are author and 
>> sponsor of "IEP-40" and we found out that point 2 in it is no longer 
>> relevant and it can be removed.
>>  I suggest implementing point 3, since it may be independent of point 1. 
>> Also, the warm-up will always start after restore phase, without subscribing 
>> to events.
>>
>>  You are right this should be mentioned in the documentation and javadoc.
>>>   This means that the user's thread, which starts Ignite via
>>>   Ignition.start(), will wait for ana additional step - cache warm-up.
>>>   I think this fact has to be clearly mentioned in our documentation (at
>>>   Javadocat least) because this step can be time-consuming.
>>
>>  My suggestion for implementation:
>>  1)Adding a marker interface 
>> "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache 
>> warming;
>>  2)Set only one configuration via 
>> "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>  3)Add an internal warm-up interface that will start in [1] after [2];
>>
>>  package org.apache.ignite.internal.processors.cache.warmup;
>>
>>  import org.apache.ignite.IgniteCheckedException;
>>  import org.apache.ignite.configuration.WarmUpConfiguration;
>>  import org.apache.ignite.internal.GridKernalContext;
>>
>>  /**
>>   * Interface for warming up.
>>   */
>>  public interface WarmUpStrategy {
>>  /**
>>   * Returns configuration class for mapping to strategy.
>>   *
>>   * @return Configuration class.
>>   */
>>  Class configClass();
>>
>>  /**
>>   * Warm up.
>>   *
>>   * @param kernalCtx Kernal context.
>>   * @param cfg Warm-up configuration.
>>   * @throws IgniteCheckedException if faild.
>>   */
>>  void warmUp(GridKernalContext kernalCtx, T cfg) throws 
>> IgniteCheckedException;
>>  }
>>
>>  4)Adding an internal plugin extension for add own strategies;
>>
>>  package org.apache.ignite.internal.processors.cache.warmup;
>>
>>  import java.util.Collection;
>>  import org.apache.ignite.plugin.Extension;
>>
>>  /**
>>   * Interface for getting warm-up strategies from plugins.
>>   */
>>  public interface WarmUpStrategySupplier extends Extension {
>>  /**
>>   * Getting warm-up strategies.
>>   *
>>   * @return Warm-up strategies.
>>   */
>>  Collection strategies();
>>  }
>>
>>  5)Add a "Load all" strategy that will load everything to memory as long as 
>> there is space in it. This strategy is suitable if the persistent storage is 
>> less than RAM.
>>
>>  Any objections or comments?
>>
>>  [1] - 
>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>  [2] - 
>> 

Re: Ignite compilation fails in IntelliJ IDEA (IgniteLinkTaglet)

[Pavel Tupitsyn]

Anton, this worked, thank you!

On Mon, Aug 10, 2020 at 1:06 PM Anton Kalashnikov  wrote:

> Hi Pavel,
>
> You can do the same action in Project tree -> right button on java11 ->
> Mark directory as -> excluded
>
> --
> Best regards,
> Anton Kalashnikov
>
>
>
> 10.08.2020, 11:34, "Ivan Bessonov" :
> > Hi Pavel,
> >
> > this issue is unrelated to your problem, but yes, it wouldn't allow you
> to
> > save changes.
> > This sucks. You should remove one of those modules in your settings, they
> > point on
> > the same pom.xml, this means that this is the same module twice in your
> > settings.
> >
> > пн, 10 авг. 2020 г. в 11:24, Pavel Tupitsyn :
> >
> >>  Ivan,
> >>
> >>  Thanks for the suggestion. Unfortunately, it does not help -
> >>  Idea does not let me apply the changes:
> >>
> >>  `Content root "/home/pavel/w/ignite" is defined for modules
> "apache-ignite"
> >>  and "ignite".
> >>  Two modules in a project cannot share the same content root.`
> >>
> >>  Clearly I'm doing something wrong - maybe I'm importing the project
> into
> >>  Idea in a wrong way?
> >>  Or should I use a different JDK? Which version is best for Ignite
> >>  development right now?
> >>  (I'm using OpenJDK 8 just out of habit)
> >>
> >>  On Mon, Aug 10, 2020 at 10:02 AM Ivan Bessonov 
> >>  wrote:
> >>
> >>  > Hi Pavel,
> >>  >
> >>  > please go to "Project Structure | Project Settings | Modules",
> >>  > find module "ignite-tools", open tab "Sources" and mark folder
> >>  > "src/main/java11" as "excluded". Should help.
> >>  >
> >>  > This happens from time to time if you switch from a very old branch
> >>  > (like "ignite-2.5") to a fresh branch like "master".
> >>  >
> >>  > вс, 9 авг. 2020 г. в 21:18, Pavel Tupitsyn :
> >>  >
> >>  > > Igniters,
> >>  > >
> >>  > > The project does not seem to compile in IDEA:
> >>  > > there are two IgniteLinkTaglet versions for Java 8 and Java 9+,
> >>  > > and both files get picked up by the IDE for some reason, resulting
> >>  > > in build errors.
> >>  > >
> >>  > > I've done all the usual things (fresh clone, invalidate caches).
> >>  > > java-8 profile is enabled, java-9+ disabled, only JDK 8 is
> installed.
> >>  > > Maven build is fine, only IDEA gives me errors.
> >>  > >
> >>  > > I've seen some people just delete one of the IgniteLinkTaglet
> files,
> >>  > > this works for me too but is quite inconvenient.
> >>  > > Is there any trick to this?
> >>  > >
> >>  >
> >>  >
> >>  > --
> >>  > Sincerely yours,
> >>  > Ivan Bessonov
> >>  >
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
>

Re: Ignite compilation fails in IntelliJ IDEA (IgniteLinkTaglet)

[Anton Kalashnikov]

Hi Pavel,

You can do the same action in Project tree -> right button on java11 -> Mark 
directory as -> excluded

-- 
Best regards,
Anton Kalashnikov



10.08.2020, 11:34, "Ivan Bessonov" :
> Hi Pavel,
>
> this issue is unrelated to your problem, but yes, it wouldn't allow you to
> save changes.
> This sucks. You should remove one of those modules in your settings, they
> point on
> the same pom.xml, this means that this is the same module twice in your
> settings.
>
> пн, 10 авг. 2020 г. в 11:24, Pavel Tupitsyn :
>
>>  Ivan,
>>
>>  Thanks for the suggestion. Unfortunately, it does not help -
>>  Idea does not let me apply the changes:
>>
>>  `Content root "/home/pavel/w/ignite" is defined for modules "apache-ignite"
>>  and "ignite".
>>  Two modules in a project cannot share the same content root.`
>>
>>  Clearly I'm doing something wrong - maybe I'm importing the project into
>>  Idea in a wrong way?
>>  Or should I use a different JDK? Which version is best for Ignite
>>  development right now?
>>  (I'm using OpenJDK 8 just out of habit)
>>
>>  On Mon, Aug 10, 2020 at 10:02 AM Ivan Bessonov 
>>  wrote:
>>
>>  > Hi Pavel,
>>  >
>>  > please go to "Project Structure | Project Settings | Modules",
>>  > find module "ignite-tools", open tab "Sources" and mark folder
>>  > "src/main/java11" as "excluded". Should help.
>>  >
>>  > This happens from time to time if you switch from a very old branch
>>  > (like "ignite-2.5") to a fresh branch like "master".
>>  >
>>  > вс, 9 авг. 2020 г. в 21:18, Pavel Tupitsyn :
>>  >
>>  > > Igniters,
>>  > >
>>  > > The project does not seem to compile in IDEA:
>>  > > there are two IgniteLinkTaglet versions for Java 8 and Java 9+,
>>  > > and both files get picked up by the IDE for some reason, resulting
>>  > > in build errors.
>>  > >
>>  > > I've done all the usual things (fresh clone, invalidate caches).
>>  > > java-8 profile is enabled, java-9+ disabled, only JDK 8 is installed.
>>  > > Maven build is fine, only IDEA gives me errors.
>>  > >
>>  > > I've seen some people just delete one of the IgniteLinkTaglet files,
>>  > > this works for me too but is quite inconvenient.
>>  > > Is there any trick to this?
>>  > >
>>  >
>>  >
>>  > --
>>  > Sincerely yours,
>>  > Ivan Bessonov
>>  >
>
> --
> Sincerely yours,
> Ivan Bessonov

Re: Ignite compilation fails in IntelliJ IDEA (IgniteLinkTaglet)

[Ivan Bessonov]

Hi Pavel,

this issue is unrelated to your problem, but yes, it wouldn't allow you to
save changes.
This sucks. You should remove one of those modules in your settings, they
point on
the same pom.xml, this means that this is the same module twice in your
settings.

пн, 10 авг. 2020 г. в 11:24, Pavel Tupitsyn :

> Ivan,
>
> Thanks for the suggestion. Unfortunately, it does not help -
> Idea does not let me apply the changes:
>
> `Content root "/home/pavel/w/ignite" is defined for modules "apache-ignite"
> and "ignite".
> Two modules in a project cannot share the same content root.`
>
> Clearly I'm doing something wrong - maybe I'm importing the project into
> Idea in a wrong way?
> Or should I use a different JDK? Which version is best for Ignite
> development right now?
> (I'm using OpenJDK 8 just out of habit)
>
>
> On Mon, Aug 10, 2020 at 10:02 AM Ivan Bessonov 
> wrote:
>
> > Hi Pavel,
> >
> > please go to "Project Structure | Project Settings | Modules",
> > find module "ignite-tools", open tab "Sources" and mark folder
> > "src/main/java11" as "excluded". Should help.
> >
> > This happens from time to time if you switch from a very old branch
> > (like "ignite-2.5") to a fresh branch like "master".
> >
> > вс, 9 авг. 2020 г. в 21:18, Pavel Tupitsyn :
> >
> > > Igniters,
> > >
> > > The project does not seem to compile in IDEA:
> > > there are two IgniteLinkTaglet versions for Java 8 and Java 9+,
> > > and both files get picked up by the IDE for some reason, resulting
> > > in build errors.
> > >
> > > I've done all the usual things (fresh clone, invalidate caches).
> > > java-8 profile is enabled, java-9+ disabled, only JDK 8 is installed.
> > > Maven build is fine, only IDEA gives me errors.
> > >
> > > I've seen some people just delete one of the IgniteLinkTaglet files,
> > > this works for me too but is quite inconvenient.
> > > Is there any trick to this?
> > >
> >
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
> >
>


-- 
Sincerely yours,
Ivan Bessonov

Re: Ignite compilation fails in IntelliJ IDEA (IgniteLinkTaglet)

[Pavel Tupitsyn]

Ivan,

Thanks for the suggestion. Unfortunately, it does not help -
Idea does not let me apply the changes:

`Content root "/home/pavel/w/ignite" is defined for modules "apache-ignite"
and "ignite".
Two modules in a project cannot share the same content root.`

Clearly I'm doing something wrong - maybe I'm importing the project into
Idea in a wrong way?
Or should I use a different JDK? Which version is best for Ignite
development right now?
(I'm using OpenJDK 8 just out of habit)


On Mon, Aug 10, 2020 at 10:02 AM Ivan Bessonov 
wrote:

> Hi Pavel,
>
> please go to "Project Structure | Project Settings | Modules",
> find module "ignite-tools", open tab "Sources" and mark folder
> "src/main/java11" as "excluded". Should help.
>
> This happens from time to time if you switch from a very old branch
> (like "ignite-2.5") to a fresh branch like "master".
>
> вс, 9 авг. 2020 г. в 21:18, Pavel Tupitsyn :
>
> > Igniters,
> >
> > The project does not seem to compile in IDEA:
> > there are two IgniteLinkTaglet versions for Java 8 and Java 9+,
> > and both files get picked up by the IDE for some reason, resulting
> > in build errors.
> >
> > I've done all the usual things (fresh clone, invalidate caches).
> > java-8 profile is enabled, java-9+ disabled, only JDK 8 is installed.
> > Maven build is fine, only IDEA gives me errors.
> >
> > I've seen some people just delete one of the IgniteLinkTaglet files,
> > this works for me too but is quite inconvenient.
> > Is there any trick to this?
> >
>
>
> --
> Sincerely yours,
> Ivan Bessonov
>

Re: [DISCUSSION] Cache warmup

[ткаленко кирилл]

Hi, Stan!

As a result of personal correspondence I realized that you are right about 
making changes:
1)Move warm-up configuration to 
org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
2)Start warming up for each region sequentially;
3)Improving warm-up interface:

package org.apache.ignite.internal.processors.cache.warmup;

import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
import org.apache.ignite.internal.processors.cache.persistence.DataRegion;

/**
 * Interface for warming up.
 */
public interface WarmUpStrategy {
/**
 * Returns configuration class for mapping to strategy.
 *
 * @return Configuration class.
 */
Class configClass();

/**
 * Warm up.
 *
 * @param kernalCtx Kernal context.
 * @param cfg   Warm-up configuration.
 * @param regionData region.
 * @throws IgniteCheckedException if faild.
 */
void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws 
IgniteCheckedException;

/**
 * Closing warm up.
 *
 * @throws IgniteCheckedException if faild.
 */
void close() throws IgniteCheckedException;
}

4)Add a command to "control.sh", to stop current warm-up and cancel all others: 
--warm-up stop
5)The "load all" strategy will work as long as there is enough RAM and index 
pages will also take priority.

04.08.2020, 13:29, "ткаленко кирилл" :
> Hi, Slava!
>
> Thank you for looking at the offer and making fair comments.
>
> I personally discussed with Anton and Alexey because they are author and 
> sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant 
> and it can be removed.
> I suggest implementing point 3, since it may be independent of point 1. Also, 
> the warm-up will always start after restore phase, without subscribing to 
> events.
>
> You are right this should be mentioned in the documentation and javadoc.
>>  This means that the user's thread, which starts Ignite via
>>  Ignition.start(), will wait for ana additional step - cache warm-up.
>>  I think this fact has to be clearly mentioned in our documentation (at
>>  Javadocat least) because this step can be time-consuming.
>
> My suggestion for implementation:
> 1)Adding a marker interface 
> "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache 
> warming;
> 2)Set only one configuration via 
> "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
> 3)Add an internal warm-up interface that will start in [1] after [2];
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.configuration.WarmUpConfiguration;
> import org.apache.ignite.internal.GridKernalContext;
>
> /**
>  * Interface for warming up.
>  */
> public interface WarmUpStrategy {
> /**
>  * Returns configuration class for mapping to strategy.
>  *
>  * @return Configuration class.
>  */
> Class configClass();
>
> /**
>  * Warm up.
>  *
>  * @param kernalCtx Kernal context.
>  * @param cfg Warm-up configuration.
>  * @throws IgniteCheckedException if faild.
>  */
> void warmUp(GridKernalContext kernalCtx, T cfg) throws 
> IgniteCheckedException;
> }
>
> 4)Adding an internal plugin extension for add own strategies;
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import java.util.Collection;
> import org.apache.ignite.plugin.Extension;
>
> /**
>  * Interface for getting warm-up strategies from plugins.
>  */
> public interface WarmUpStrategySupplier extends Extension {
> /**
>  * Getting warm-up strategies.
>  *
>  * @return Warm-up strategies.
>  */
> Collection strategies();
> }
>
> 5)Add a "Load all" strategy that will load everything to memory as long as 
> there is space in it. This strategy is suitable if the persistent storage is 
> less than RAM.
>
> Any objections or comments?
>
> [1] - 
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
> [2] - 
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>
> 27.07.2020, 16:48, "Вячеслав Коптилин" :
>>  Hello Kirill,
>>
>>  Thanks a lot for driving this activity. If I am not mistaken, this
>>  discussion relates to IEP-40.
>>
>>>   I suggest adding a warmup phase after recovery here [1] after [2], before
>>
>>  discovery.
>>  This means that the user's thread, which starts Ignite via
>>  Ignition.start(), will wait for ana additional step - cache warm-up.
>>  I think this fact has to be clearly mentioned in our documentation (at
>>  Javadocat least) because this step can be time-consuming.
>>
>>>   I suggest adding a new interface:
>>
>>  I would change it a bit. First of all, it would be nice to place this
>>  

Re: Ignite compilation fails in IntelliJ IDEA (IgniteLinkTaglet)

[Ivan Bessonov]

Hi Pavel,

please go to "Project Structure | Project Settings | Modules",
find module "ignite-tools", open tab "Sources" and mark folder
"src/main/java11" as "excluded". Should help.

This happens from time to time if you switch from a very old branch
(like "ignite-2.5") to a fresh branch like "master".

вс, 9 авг. 2020 г. в 21:18, Pavel Tupitsyn :

> Igniters,
>
> The project does not seem to compile in IDEA:
> there are two IgniteLinkTaglet versions for Java 8 and Java 9+,
> and both files get picked up by the IDE for some reason, resulting
> in build errors.
>
> I've done all the usual things (fresh clone, invalidate caches).
> java-8 profile is enabled, java-9+ disabled, only JDK 8 is installed.
> Maven build is fine, only IDEA gives me errors.
>
> I've seen some people just delete one of the IgniteLinkTaglet files,
> this works for me too but is quite inconvenient.
> Is there any trick to this?
>


-- 
Sincerely yours,
Ivan Bessonov

[jira] [Created] (IGNITE-13344) DummyVectorizer fails to extract label for coordinate with value "0.0" when backed by sparse vector

[Thilo-Alexander Ginkel (Jira)]

Thilo-Alexander Ginkel created IGNITE-13344:
---

 Summary: DummyVectorizer fails to extract label for coordinate 
with value "0.0" when backed by sparse vector
 Key: IGNITE-13344
 URL: https://issues.apache.org/jira/browse/IGNITE-13344
 Project: Ignite
  Issue Type: Bug
  Components: ml
Affects Versions: 2.8.1
Reporter: Thilo-Alexander Ginkel


Given: A labeled DummyVectorizer:

 
{code:java}
new DummyVectorizer()
 .exclude(excludeCoordinates.stream().map(coord -> vectorLength + 
coord).toArray(Integer[]::new))
 .labeled(labelCoord);
{code}
{{When extracting the label, the call hierarchy eventually ends up at 
org.apache.ignite.ml.dataset.feature.extractor.impl.DummyVectorizer#feature, 
which returns null for val.getRaw when val is a sparse vector with the element 
at the requested label coordinate being 0.0. This causes the training job to 
fail (which expects a non-null label):}}
{noformat}
org.apache.ignite.IgniteException: Remote job threw user exception (override or 
implement ComputeTask.result(..) method if you would like to have automatic 
failover for this exception): nullorg.apache.ignite.IgniteException: Remote job 
threw user exception (override or implement ComputeTask.result(..) method if 
you would like to have automatic failover for this exception): null at 
org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:961)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:809)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:659)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) 
~[ignite-core-2.8.1.jar:2.8.1] at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
 ~[na:na] at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
 ~[na:na] at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]Caused by: 
org.apache.ignite.IgniteException: null at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1858)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
 ~[ignite-core-2.8.1.jar:2.8.1] at 
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
 ~[ignite-core-2.8.1.jar:2.8.1] ... 5 common frames omittedCaused by: 
java.lang.NullPointerException: null at 
org.apache.ignite.ml.dataset.impl.bootstrapping.BootstrappedDatasetBuilder.build(BootstrappedDatasetBuilder.java:91)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 
org.apache.ignite.ml.dataset.impl.bootstrapping.BootstrappedDatasetBuilder.build(BootstrappedDatasetBuilder.java:41)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 
org.apache.ignite.ml.dataset.impl.cache.util.ComputeUtils.lambda$getData$4(ComputeUtils.java:239)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 
org.apache.ignite.ml.dataset.impl.cache.util.PartitionDataStorage.lambda$computeDataIfAbsent$1(PartitionDataStorage.java:56)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
 ~[na:na] at 
org.apache.ignite.ml.dataset.impl.cache.util.PartitionDataStorage.computeDataIfAbsent(PartitionDataStorage.java:56)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 
org.apache.ignite.ml.dataset.impl.cache.util.ComputeUtils.getData(ComputeUtils.java:209)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 
org.apache.ignite.ml.dataset.impl.cache.CacheBasedDataset.lambda$compute$c7cefc59$1(CacheBasedDataset.java:172)
 ~[ignite-ml-2.8.1.jar:2.8.1] at 

Re: Apache Ignite 3.0

[Petr Ivanov]

Hi, Val!
Thanks for your efforts on this endeavour!


I would like to suggest deliveries changes in Apache Ignite 3.0:
 — modularised  binary delivery — single minimal binary for starting Ignite and 
all other modules and parts of the project (benchmarks, examples, etc.) packed 
in their own binary which can be added via custom dependency management tool 
(i.e. modules.sh)
 — same distribution for RPM and DEB packages but with modules packed as 
separate ones (PHP for example)
 — separate thin client release cycle with custom versioning
Possibly, we can we add additional section to the document you introduced for 
this part.

Also, it seems that full JDK11 support (including building) would be a huge 
milestone and a sign of healthy modern project that tends to be on the verge of 
mainstream technologies and not the stockpile of legacy leftovers (fully 
support Iliya in removing all that was deprecated and/or marked as unused 
anymore).


> On 8 Aug 2020, at 02:00, Valentin Kulichenko  
> wrote:
> 
> Igniters,
> 
> I've created the page:
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
> 
> That's not everything I have in mind, but I believe there is already a lot
> to talk about :)
> 
> Please take a look let me know if you have any concerns, objections, or
> questions. Once we reach the consensus on the proposed changes, I will
> start creating tickets in Jira and a more detailed plan.
> 
> -Val
> 
> On Thu, Aug 6, 2020 at 6:28 PM Saikat Maitra 
> wrote:
> 
>> Hi Denis, Val
>> 
>> Thank you for your reply and really appreciate it. It will be very cool to
>> be able to connect and plan release together and learn more about Ignite in
>> the process :)
>> 
>> Regards
>> Saikat
>> 
>> 
>> 
>> On Thu, Aug 6, 2020 at 7:12 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> 
>>> Hi Saikat,
>>> 
>>> That surely is a great idea. We will work together with Denis on setting
>>> this up in the nearest future.
>>> 
>>> -Val
>>> 
>>> On Thu, Aug 6, 2020 at 10:21 AM Denis Magda  wrote:
>>> 
 Saikat,
 
 Fully support your idea on a virtual meetup! Once Val collects and
>>> outlines
 the main changes with directions on wiki, we’ll go ahead and schedule
>> the
 meetup to talk things out in a bit more detail. We’ll use our new
>> Virtual
 Ignite Meetup group for that inviting both Ignite contributors and
 application developers.
 
 Denis
 
 On Thursday, August 6, 2020, Saikat Maitra 
 wrote:
 
> Hi Valentin
> 
> Thank you for sharing and starting the thread. I am thinking if it
>> will
 be
> a good idea to have a virtual meet setup to discuss on the release
> planning.
> 
> It will help to learn more individual features to be added and also
>> to
> understand about features that have been deprecated and scheduled for
> removal in Ignite 3.0 release. Also it will help community member to
> connect in real time and ask questions and share feedback.
> 
> Regards,
> Saikat
> 
> On Thu, Aug 6, 2020 at 3:51 AM Ilya Kasnacheev <
 ilya.kasnach...@gmail.com>
> wrote:
> 
>> Hello!
>> 
>> I hope to see Apache Ignite release 3.0 as API trimming release.
>> Let
>>> us
>> correct external and internal APIs for which we have better ideas
>>> now,
 as
>> well as remove old and deprecated code.
>> 
>> We may also introduce new configuration mechanisms and user-facing
>>> API
>> (such as cache-less native SQL queries), but this we could
>> prototype
> before
>> starting the 3.0 task.
>> 
>> I will advise against targeting large new features at 3.0. They can
>>> be
>> added in subsequent point releases, whereas we can't really remove
>> or
>> remodel stuff in point releases.
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> чт, 6 авг. 2020 г. в 03:54, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com>:
>> 
>>> Igniters,
>>> 
>>> I would like to kick off a discussion regarding Ignite 3.0.
>> Ignite
 2.0
>>> exists for more than 3 years now and we've already collected a
>> significant
>>> list [1] of changes that we would like to have, but cannot
>>> implement
>>> without breaking compatibility.
>>> 
>>> I think it's time to start planning for the next major release
>> and
>>> discussing what should be included. I've already gathered some
>> information
>>> and feedback, and have some thoughts on how to approach this. In
>>> the
> next
>>> few days, I will put everything into a Wiki page and will share
>> it
 once
>>> this is done. Stay tuned!
>>> 
>>> I'm willing to drive the 3.0 activities going forward as well.
>>> 
>>> In the meantime, if there are any immediate thoughts or ideas,
>>> please
>> feel
>>> free to join the thread and share them.
>>> 
>>> [1]
>>> 
>>>