Re: [VOTE] Ignite Packaging IEP

[Алексей]

+1

On Mon, 22 Aug 2022 at 17:07 Mikhail Pochatkin 
wrote:

> Hi Igniters,
>
> I want to start a voting process about several points from IEP-93 [1].
>
> 1. Switch Apache Ignite build system to Gradle as default.
> 2. Consequently, after changing the build system to Gradle we need to add
> new third party dependencies to several Gradle plugins. This is list of new
> complete time deps:
> openapiGenerator = "org.openapi.generator:5.4.0"
> javacc = "com.intershop.gradle.javacc:4.0.1"
> launch4j = "edu.sc.seis.launch4j:2.5.3"
> shadow = "com.github.johnrengelman.shadow:7.1.2"
> cmake = "io.github.tomtzook.gradle-cmake:1.0.0"
> jreleaser =  "org.jreleaser:1.1.0"
> 3. Agreement on all target package formats for Apache Ignite distributions.
>
> The vote is formal, see voting guidelines [3].
>
> +1 - to accept all points above
> 0 - don’t care either way
> -1 - DO NOT accept (explain why)
>
> This vote will be open for at least 4 days till Friday Aug 26, 2022,
> 22:00 Moscow TZ.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-93%3A+Ignite+Packaging
> [2] https://www.apache.org/foundation/voting.html
>

Re: [VOTE] Add micronaut dependency to Ignite 3 (reopened)

[Алексей]

+1

ср, 1 июн. 2022 г. в 12:19, Alexander Lapin :

> +1
>
> ср, 1 июн. 2022 г. в 12:14, ткаленко кирилл :
>
> > + 1
> >
>

Re: Apache Ignite 2.11

[Алексей Гидаспов]

:scope:

Here is a list [1] of 195 tickets, i picked for Apache Ignite 2.11 release

Let's start discussion

[1] 
https://issues.apache.org/jira/browse/IGNITE-14781?jql=project%20%3D%20IGNITE%20%20AND%20status%20%20%3D%20Resolved%20and%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

On 2021/05/13 14:27:33, Dmitry Pavlov  wrote: 
> Hi Alexey,
> 
> :pmc to sign distribution:
> Formally, only PMC member can be a release manager, so part of activities 
> should be picked up by committer and PMC member. If noone else want to help 
> you here, I would.
> 
> :timeline:
> And, could you estimate all phases of the release. 
> 
> :scope:
> I guess we can just pick up master current state and release it. But if 
> someone has some ideas if we should wait for particular feature to be 
> completed before branch divergence/release candidate build, please let know.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> On 2021/05/13 09:45:02, Алексей Гидаспов  wrote: 
> > Dear Ignite Community!
> > 
> > I suggest starting Apache Ignite 2.11 release activities. Also suggest
> > myself to be a release manager for this release. I plan release at the end
> > of june 2021.
> > 
> > ___
> > 
> > Best Regards,
> > Alexey Gidaspov
> > 
> 

Re: Edit access to Apache Ignite wiki

[Алексей Гидаспов]

Checked, it's ok. Thank you. 

On 2021/06/03 08:27:07, Dmitry Pavlov  wrote: 
> Alexey,
> 
> I've granted you edit permissions.
> Please check availability.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> On 2021/06/03 08:24:03, ��  
>  wrote: 
> > My login is: agidaspov
> > E-mail: olive.c...@gmail.com
> > 
> > On 2021/06/03 08:17:15, Dmitry Pavlov  wrote: 
> > > Can you please sign in to cwiki and share your login?
> > > 
> > > wiki admins (PMC members) could grant access in that case.
> > > 
> > > On 2021/06/03 07:11:54, Алексей Гидаспов  wrote: 
> > > > Hi, Igniters!
> > > > 
> > > > As a release manager of 2.11 Apache Ignite release I need edit access to
> > > > cwiki.apache.org Apache Ignite space. Please tell me, how can I obtain 
> > > > it?
> > > > 
> > > 
> > 
> 

Re: Edit access to Apache Ignite wiki

[Алексей Гидаспов]

My login is: agidaspov
E-mail: olive.c...@gmail.com

On 2021/06/03 08:17:15, Dmitry Pavlov  wrote: 
> Can you please sign in to cwiki and share your login?
> 
> wiki admins (PMC members) could grant access in that case.
> 
> On 2021/06/03 07:11:54, Алексей Гидаспов  wrote: 
> > Hi, Igniters!
> > 
> > As a release manager of 2.11 Apache Ignite release I need edit access to
> > cwiki.apache.org Apache Ignite space. Please tell me, how can I obtain it?
> > 
> 

Edit access to Apache Ignite wiki

[Алексей Гидаспов]

Hi, Igniters!

As a release manager of 2.11 Apache Ignite release I need edit access to
cwiki.apache.org Apache Ignite space. Please tell me, how can I obtain it?

Re: Apache Ignite 2.11

[Алексей Гидаспов]

Hi, Dmitriy!

I suggest this timeline

Scope Freeze: May 31, 2021
Code Freeze: 14 June, 2021
Voting Date: 21 June, 2021
Release Date: 28 June, 2021

пт, 14 мая 2021 г. в 11:32, Maksim Timonin :

> Hi, I'd like to include Phase 1 of index queries feature [1] to the next AI
> release. Currently there are 2 tickets in progress ([2], [3]), and they
> will be ready for review next week. What is a time of code freeze if it's
> going to release AI 2.11 at the end of June?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-71%3A+Public+API+for+secondary+index+search
> [2] https://issues.apache.org/jira/browse/IGNITE-14699
> [3] https://issues.apache.org/jira/browse/IGNITE-14703
>
> On Thu, May 13, 2021 at 5:27 PM Dmitry Pavlov  wrote:
>
> > Hi Alexey,
> >
> > :pmc to sign distribution:
> > Formally, only PMC member can be a release manager, so part of activities
> > should be picked up by committer and PMC member. If noone else want to
> help
> > you here, I would.
> >
> > :timeline:
> > And, could you estimate all phases of the release.
> >
> > :scope:
> > I guess we can just pick up master current state and release it. But if
> > someone has some ideas if we should wait for particular feature to be
> > completed before branch divergence/release candidate build, please let
> know.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > On 2021/05/13 09:45:02, Алексей Гидаспов  wrote:
> > > Dear Ignite Community!
> > >
> > > I suggest starting Apache Ignite 2.11 release activities. Also suggest
> > > myself to be a release manager for this release. I plan release at the
> > end
> > > of june 2021.
> > >
> > > ___
> > >
> > > Best Regards,
> > > Alexey Gidaspov
> > >
> >
>

Apache Ignite 2.11

[Алексей Гидаспов]

Dear Ignite Community!

I suggest starting Apache Ignite 2.11 release activities. Also suggest
myself to be a release manager for this release. I plan release at the end
of june 2021.

___

Best Regards,
Alexey Gidaspov

Hello all

[Алексей Высотин]

Hello Ignite Community,

My name is Alexey and I'm a Java and Scala developer with a current focus
on Big Data stack like Spark and Hadoop.
I would like to take part in the Apache Ignite project and make a
contribution to it.

I'm thinking to start with some easy fix or bugfix to get more
acquainted with the code.

Could someone give me access to Jira, please? So that I will be able to
book a ticket.


My Jira Username is Tesio.

Thank you in advance.

Best Regards,

Alex Vysotin

Re: Re[2]: [VOTE] Apache Ignite PMC Chair

[Алексей Платонов]

+1 for Dmitry Pavlov

чт, 31 окт. 2019 г. в 01:23, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> +1 Dmitry Pavlov (binding)
>
> On Wed, Oct 30, 2019 at 12:55 PM Alex Plehanov 
> wrote:
>
> > + 1 Dmitry Pavlov
> >
> > ср, 30 окт. 2019 г. в 20:50, Pavel Kovalenko :
> >
> > > +1 for Dmitry Pavlov
> > >
> > > ср, 30 окт. 2019 г. в 18:46, Alexei Scherbakov <
> > > alexey.scherbak...@gmail.com
> > > >:
> > >
> > > > +1 for Dmitry Pavlov
> > > >
> > > > ср, 30 окт. 2019 г. в 18:22, aealexsandrov  >:
> > > >
> > > > > +1 Alexey Goncharuk
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> >
>

Re: [ML] Distributed metrics computation

[Алексей Платонов]

Hi, Vyacheslav,
Thanks for the advice. Actually, we already have the MapReduce approach
implementation in ML dataset and this implementation is based on compute
task. So, I think that I just can to reuse this solution.

Best regards,
Alexey Platonov

вт, 10 сент. 2019 г., 14:27 Vyacheslav Daradur :

> Hi, Alexey,
>
> I agree that Map-Reduce on demand looks more promising solution.
> We can use Compute tasks for implementation.
> 'Map' phase can be tunned to process data by some trigger (dataset
> update?) on ContiniousQuery manner and call 'Reduce' (with some
> cache?) on demand.
>
>
> On Tue, Sep 10, 2019 at 2:09 PM Алексей Платонов 
> wrote:
> >
> > I mean metrics for model evaluation like Accuracy or Precision/Recall for
> > ML models. It isn't same as system metrics (like throughput). Such
> metrics
> > should be computed over a test set after model training. if it is
> > interesting for you, please, have a look at this material:
> > https://en.wikipedia.org/wiki/Precision_and_recall . It's just homonymy
> > between machine learning metrics and system metrics. We can't compute
> > ML-metrics via Zabbix for example.
> >
> > Best regards,
> > Alexey Platonov
> >
> > вт, 10 сент. 2019 г. в 13:52, Nikolay Izhikov :
> >
> > > Hello, Alexey.
> > >
> > > Why do we need distributed metrics in the first place?
> > > It seems, there are many metric processing system out there:
> Prometheus,
> > > Zabbix, Splunk, etc.
> > >
> > > Each of then can aggregate metrics in many ways.
> > >
> > > I think, we should not use Ignite as an metrics aggregation system.
> > >
> > > What do you think?
> > >
> > > В Вт, 10/09/2019 в 13:08 +0300, Алексей Платонов пишет:
> > > > Hi Igniters!
> > > > I've been working on a prototype of distributed metrics computation
> for
> > > > ML-models. Unfortunately, we don't have an ability to compute
> metrics in
> > > a
> > > > distributed manner, so, it leads to gathering metric statistics to
> client
> > > > node via ScanQuery and all flow of vectors from partitions will be
> sent
> > > to
> > > > a client. I want to avoid such behavior and I propose the framework
> for
> > > > metrics computation using MapReduce approach based on an aggregation
> of
> > > > statistics for metrics.
> > > >
> > > > I prepared an issue in Apache Jira for this:
> > > > https://issues.apache.org/jira/browse/IGNITE-12155
> > > > Also, I prepared PR for it:
> https://github.com/apache/ignite/pull/6857
> > > > Currently, the work on this framework is still running but I'm going
> to
> > > > prepare full PR during this week.
> > > >
> > > > By this email, I want to start a discussion about this idea.
> > > >
> > > > Best regards,
> > > > Alexey Platonov
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: [ML] Distributed metrics computation

[Алексей Платонов]

I mean metrics for model evaluation like Accuracy or Precision/Recall for
ML models. It isn't same as system metrics (like throughput). Such metrics
should be computed over a test set after model training. if it is
interesting for you, please, have a look at this material:
https://en.wikipedia.org/wiki/Precision_and_recall . It's just homonymy
between machine learning metrics and system metrics. We can't compute
ML-metrics via Zabbix for example.

Best regards,
Alexey Platonov

вт, 10 сент. 2019 г. в 13:52, Nikolay Izhikov :

> Hello, Alexey.
>
> Why do we need distributed metrics in the first place?
> It seems, there are many metric processing system out there: Prometheus,
> Zabbix, Splunk, etc.
>
> Each of then can aggregate metrics in many ways.
>
> I think, we should not use Ignite as an metrics aggregation system.
>
> What do you think?
>
> В Вт, 10/09/2019 в 13:08 +0300, Алексей Платонов пишет:
> > Hi Igniters!
> > I've been working on a prototype of distributed metrics computation for
> > ML-models. Unfortunately, we don't have an ability to compute metrics in
> a
> > distributed manner, so, it leads to gathering metric statistics to client
> > node via ScanQuery and all flow of vectors from partitions will be sent
> to
> > a client. I want to avoid such behavior and I propose the framework for
> > metrics computation using MapReduce approach based on an aggregation of
> > statistics for metrics.
> >
> > I prepared an issue in Apache Jira for this:
> > https://issues.apache.org/jira/browse/IGNITE-12155
> > Also, I prepared PR for it: https://github.com/apache/ignite/pull/6857
> > Currently, the work on this framework is still running but I'm going to
> > prepare full PR during this week.
> >
> > By this email, I want to start a discussion about this idea.
> >
> > Best regards,
> > Alexey Platonov
>

[ML] Distributed metrics computation

[Алексей Платонов]

Hi Igniters!
I've been working on a prototype of distributed metrics computation for
ML-models. Unfortunately, we don't have an ability to compute metrics in a
distributed manner, so, it leads to gathering metric statistics to client
node via ScanQuery and all flow of vectors from partitions will be sent to
a client. I want to avoid such behavior and I propose the framework for
metrics computation using MapReduce approach based on an aggregation of
statistics for metrics.

I prepared an issue in Apache Jira for this:
https://issues.apache.org/jira/browse/IGNITE-12155
Also, I prepared PR for it: https://github.com/apache/ignite/pull/6857
Currently, the work on this framework is still running but I'm going to
prepare full PR during this week.

By this email, I want to start a discussion about this idea.

Best regards,
Alexey Platonov

Re: [ML] Deployment of user-defined preprocessors

[Алексей Платонов]

Hi, Igniters!
I've prepared a PR with fixes to correct peer class loading for ML-lambdas.
Here it is: https://github.com/apache/ignite/pull/6853

It will be great if someone reviews my PR. Alexey, Yuriy, could you please
to see this contribution.

Thanks!

Best regards.
Alexey Platonov

ср, 5 июн. 2019 г. в 17:20, Yuriy Babak :

> Alexey,
>
> This is a cool change, do you create a ticket for it?
>
> If no I can create one.
>
> Best regards,
> Yuriy Babak
>
>
> пт, 31 мая 2019 г. в 14:20, Алексей Платонов :
>
> > Hi, Igniters!
> > Currently we don't have an ability to deploy automatically user-defined
> > preprocessors and vectorizers. Client's code should be deployed manually
> to
> > Ignite server nodes.
> >
> > I have an idea how to fix it. If we pass user's classloader and one of
> > user-defined classes from fit-level to
> > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > GridPeerDeployAware interface to send informtation about this classloader
> > to server nodes.
> >
> > To support this ability we can define interfaces like these:
> >
> > public interface DeployableObject {
> > public List getDependencies();
> > }
> >
> > and
> >
> > public interface DeployingContext {
> > public Class userClass();
> > public ClassLoader clientClassLoader();
> > }
> >
> > DeployableObject will be mark for our ignite-ml final classes like
> trainers
> > or concrete preprocessors and it can be able to return all dependencies
> > that should be deployed to server nodes if it's needed. If these
> > dependencies are DeployableObjects too then depenndencies will be
> unfolded
> > recursively. Classes that isn't defined as DeployableObject will be
> > recognized as user-defined (NOTE: all leaf classes in our hierarchy will
> be
> > DeployableObject).
> >
> > This list of DeployableObjects will be user for define user class loader
> > and one of these objects will be used for passing to GridPeerDeployAware.
> >
> > So, this logic allows to pass user-defined Preprocessors and Vectorizers
> to
> > training algorithms and pipelines.
> >
> > What do you think?
> >
> > Sincerely
> > Alexey Platonov
> >
>

XA transactions support

[Алексей Самойленко]

Hello, it would be nice to have a subj in Ignite.
Thx

Re: [ML] Deployment of user-defined preprocessors

[Алексей Платонов]

No, I won't change them.
In context of this architecture we should pass learning environment to
ComputeUtils and this concerns just DatasetBuilders and ComputeUtils APIs.
APIs of Preprocessors and Trainers won't be changed. We just set
DeploymentContext in fit method but it doens't require to change APIs of
these classes.

Sincerely
Alexey Platonov

пт, 31 мая 2019 г. в 14:51, Alexey Zinoviev :

> It sounds great, could you please explain what are you going to change in
> the main Vectorizer/Preprocessor API to support it?
>
> пт, 31 мая 2019 г. в 14:20, Алексей Платонов :
>
> > Hi, Igniters!
> > Currently we don't have an ability to deploy automatically user-defined
> > preprocessors and vectorizers. Client's code should be deployed manually
> to
> > Ignite server nodes.
> >
> > I have an idea how to fix it. If we pass user's classloader and one of
> > user-defined classes from fit-level to
> > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > GridPeerDeployAware interface to send informtation about this classloader
> > to server nodes.
> >
> > To support this ability we can define interfaces like these:
> >
> > public interface DeployableObject {
> > public List getDependencies();
> > }
> >
> > and
> >
> > public interface DeployingContext {
> > public Class userClass();
> > public ClassLoader clientClassLoader();
> > }
> >
> > DeployableObject will be mark for our ignite-ml final classes like
> trainers
> > or concrete preprocessors and it can be able to return all dependencies
> > that should be deployed to server nodes if it's needed. If these
> > dependencies are DeployableObjects too then depenndencies will be
> unfolded
> > recursively. Classes that isn't defined as DeployableObject will be
> > recognized as user-defined (NOTE: all leaf classes in our hierarchy will
> be
> > DeployableObject).
> >
> > This list of DeployableObjects will be user for define user class loader
> > and one of these objects will be used for passing to GridPeerDeployAware.
> >
> > So, this logic allows to pass user-defined Preprocessors and Vectorizers
> to
> > training algorithms and pipelines.
> >
> > What do you think?
> >
> > Sincerely
> > Alexey Platonov
> >
>

[ML] Deployment of user-defined preprocessors

[Алексей Платонов]

Hi, Igniters!
Currently we don't have an ability to deploy automatically user-defined
preprocessors and vectorizers. Client's code should be deployed manually to
Ignite server nodes.

I have an idea how to fix it. If we pass user's classloader and one of
user-defined classes from fit-level to
ComputeUtils.affinityCallWithRetries() then we wiil be able to use
GridPeerDeployAware interface to send informtation about this classloader
to server nodes.

To support this ability we can define interfaces like these:

public interface DeployableObject {
public List getDependencies();
}

and

public interface DeployingContext {
public Class userClass();
public ClassLoader clientClassLoader();
}

DeployableObject will be mark for our ignite-ml final classes like trainers
or concrete preprocessors and it can be able to return all dependencies
that should be deployed to server nodes if it's needed. If these
dependencies are DeployableObjects too then depenndencies will be unfolded
recursively. Classes that isn't defined as DeployableObject will be
recognized as user-defined (NOTE: all leaf classes in our hierarchy will be
DeployableObject).

This list of DeployableObjects will be user for define user class loader
and one of these objects will be used for passing to GridPeerDeployAware.

So, this logic allows to pass user-defined Preprocessors and Vectorizers to
training algorithms and pipelines.

What do you think?

Sincerely
Alexey Platonov

Re: [ML][DISCUSSION] The future of Vectorizer

[Алексей Платонов]

Hey, Igniters!
I prepared some PR for Serializable support in our Vectors.
Could you review this: https://github.com/apache/ignite/pull/6378 ?

чт, 28 мар. 2019 г. в 11:47, Алексей Платонов :

> Yep, I definitely agree with you.
>
> Moreover, such improvement should reduce parallel hierarchies in trainers
> and preprocessors, from this point of view preprocessor will be equal to a
> trainer. In my opinion, this improvement is very important for ml module
> because it can give a flexible hierarchy of components.
>
> I created a ticket for serializable object support in Vectors:
> https://issues.apache.org/jira/browse/IGNITE-11647
> Another related ticket to this thread:
> https://issues.apache.org/jira/browse/IGNITE-11642
>
> чт, 28 мар. 2019 г. в 11:27, Alexey Zinoviev :
>
>> Hi, Igniters
>>
>> The new functionality of building Vectors was merged to Apache Ignite in
>> the
>> next commit
>> <
>> https://github.com/apache/ignite/commit/a0a15d62a250defb0db9ec72153ee287830f6a15>
>>
>>
>> This new functionality brings to Ignite ML the new approach of building
>> vectors. But in my opinion the shouldn't constrain ourselves with narrow
>> understanding of Vector nature as an analogue of double[] array.
>>
>> I suggest to extend the Vector and Vectorizer API to support Strings and
>> another types (like Blobs, Images and etc) as a vector elements.
>>
>> It brings next advantages:
>> * gives a chance to inify the hierarchy of Preprocessing Trainers and
>> Model
>> Trainers
>> * give us a chance to implement ML algorithms working not only with
>> doubles
>> * unifies our Vectorizers as a first step in our Pipelines
>> * drops a lot of unused generics
>> * makes one simple requirement to final users: convert their data to
>> Vectors
>>
>> Join to discussion, ML-interested persons and share your opinon here!
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>
>

Re: [ML][DISCUSSION] The future of Vectorizer

[Алексей Платонов]

Yep, I definitely agree with you.

Moreover, such improvement should reduce parallel hierarchies in trainers
and preprocessors, from this point of view preprocessor will be equal to a
trainer. In my opinion, this improvement is very important for ml module
because it can give a flexible hierarchy of components.

I created a ticket for serializable object support in Vectors:
https://issues.apache.org/jira/browse/IGNITE-11647
Another related ticket to this thread:
https://issues.apache.org/jira/browse/IGNITE-11642

чт, 28 мар. 2019 г. в 11:27, Alexey Zinoviev :

> Hi, Igniters
>
> The new functionality of building Vectors was merged to Apache Ignite in
> the
> next commit
> <
> https://github.com/apache/ignite/commit/a0a15d62a250defb0db9ec72153ee287830f6a15>
>
>
> This new functionality brings to Ignite ML the new approach of building
> vectors. But in my opinion the shouldn't constrain ourselves with narrow
> understanding of Vector nature as an analogue of double[] array.
>
> I suggest to extend the Vector and Vectorizer API to support Strings and
> another types (like Blobs, Images and etc) as a vector elements.
>
> It brings next advantages:
> * gives a chance to inify the hierarchy of Preprocessing Trainers and Model
> Trainers
> * give us a chance to implement ML algorithms working not only with doubles
> * unifies our Vectorizers as a first step in our Pipelines
> * drops a lot of unused generics
> * makes one simple requirement to final users: convert their data to
> Vectors
>
> Join to discussion, ML-interested persons and share your opinon here!
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Tests for ML using binary builds

[Алексей Платонов]

Yes, sure.
Ignite ML algorithms actively use data sending between nodes and in several
cases uses per class loading mechanism.
I want to exclude failures when algorithms use unserializable data or try
to send lambdas with big context etc.
>From this point of view we can just run ML examples on a little cluster
where servers is started from binary build.

сб, 2 мар. 2019 г. в 08:39, Павлухин Иван :

> Hi Alexey,
>
> Could you please share some background? What problem are you solving
> with running tests against binary builds? Perhaps, we need something
> similar for other Ignite sub-projects as well.
>
> пт, 1 мар. 2019 г. в 19:04, Алексей Платонов :
> >
> > Hello, Igniters!
> > I would like to create several tests for ML algorithms using binary
> builds.
> > These tests should work in this way:
> > 1) Get last master (or user-defined branch) from git repository;
> > 2) Build Ignite with a release profile and create binary build;
> > 3) Run several Ignite instances from binary build;
> > 4) Run examples or synthetic tests with a training of ML algorithms and
> > inference;
> > 5) Accumulate fails statistics on some board.
> >
> > Currently, I'm working with own open repository in git that contains
> > scripts for Docker and Travis as the prototype. I want to complete these
> > tests and contribute them to Ignite.
> >
> > Should I adapt such tests for TC after prototype complete or Travis can
> be
> > reused? Maybe such a process was created for other Ignite modules and I
> can
> > use it for ML. What do you think?
> >
> > Best regards
> > Alexey Platonov.
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Tests for ML using binary builds

[Алексей Платонов]

Hello, Igniters!
I would like to create several tests for ML algorithms using binary builds.
These tests should work in this way:
1) Get last master (or user-defined branch) from git repository;
2) Build Ignite with a release profile and create binary build;
3) Run several Ignite instances from binary build;
4) Run examples or synthetic tests with a training of ML algorithms and
inference;
5) Accumulate fails statistics on some board.

Currently, I'm working with own open repository in git that contains
scripts for Docker and Travis as the prototype. I want to complete these
tests and contribute them to Ignite.

Should I adapt such tests for TC after prototype complete or Travis can be
reused? Maybe such a process was created for other Ignite modules and I can
use it for ML. What do you think?

Best regards
Alexey Platonov.

Contribution permission and ignite-2196

[Наумов Алексей]

Hello!
I would like to contribute to apache ignite and i am asking for contribution 
permission.
I would like to start with ignite-2196 to get familiar with contribution 
process. The issue Is still reproducable. I am plannig to remove the link and 
wrap it into @code directive as suggested in comments to jira issue. Is this ok?

My jira and github username: naumov-a

Aleksei Naumov

Contribution permission and ignite-2196

[Наумов Алексей]

Hello!
I would like to contribute to apache ignite and i am asking for contribution 
permission.
I would like to start with ignite-2196 to get familiar with contribution 
process. The issue Is still reproducable (broken link from documentation -  
https://ignite.apache.org/releases/latest/javadoc/index.html). Im plannig to 
remove the link and wrap it into @code directive as suggested in comments to 
jira issue. Is this ok?

Re: [ML] Metric calculation for classification models

[Алексей Платонов]

You can compute just TP (true-positive), FP, TN and FN counters and use
them to evaluate Recall, Precision, Accuracy, ect. If you want to specify
class for Pr evaluation, then you can compute Pr for first label as
TP/(TP+FP) and for second label as TN/(TN+FN) for example. After it we can
unite all one-point metrics evaluation.

In my opinion we can redesign metrics calculation and provide one-point
metrics (like Pr, Re) and integral metrics like ROC AUC where one-point
metrics can be calculated through TP,FP etc.

Maybe you should design class BinaryClassificationMetric that computes
these counters and provide methods like recall :: () -> double, precision
:: () -> double, etc.

чт, 13 дек. 2018 г. в 13:26, Yuriy Babak :

> Igniters, Alexey
>
> I want to discuss the ticket 10371 [1], currently, we calculate 4 numbers
> (true positive, true negative, false positive, false negative) for each
> "point metric" like accuracy, recall, f-score and precision for each label.
>
> So for the full score we need calculates those 4 numbers 8 times. But we
> could calculate all 8 metrics(4 for the first label and 4 for the second
> label).
>
> I suggest introducing new API "point metric" for metrics like those
> 4(accuracy, recall, f-score, and precision) and "integral metric" for
> metrics like ROC AUC [2].
>
> Any thoughts would be appreciated.
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-10371
> [2] - https://issues.apache.org/jira/browse/IGNITE-10145
>

Re: Upgrading Ignite to JCache 1.1

[Кузнецов Алексей Львович]

Hi

After Alexander create separate tickets for failed tests, everybody is 
free to fix them.

So we can proceed faster issue resolving.

Hello, Igniters.

Actually, I don't understand why we should merge in master something broken.
Currently, Ignite is not ready for JCache 1.1.

Only change I see in PR is new profile [1].
Is it required to have it to continue jcache 1.1 support implementation?

I think Alexandex can proceed with current profile and change it  to run tests 
for JCache 1.1 his own branch.

Am I miss something?

[1] https://github.com/apache/ignite/pull/4114

В Вт, 05/06/2018 в 12:50 +0300, Dmitry Pavlov пишет:

Hi Alexander,

https://issues.apache.org/jira/browse/IGNITE-8687 is 'In progress'. Is it
expected?

Nikolay, have you some time to apply patch, if it passes review?

Sincerely,
Dmitriy Pavlov

вт, 5 июн. 2018 г. в 5:09, Dmitriy Setrakyan :


Thanks, Alex! Sounds like a good plan.

On Mon, Jun 4, 2018 at 5:52 AM, Александр Меньшиков 
wrote:


Hi,
Igniters!

I have taken a look at the jcache 1.1 spec and TCK.
And I can write a brief summary of my plan to solve the task.

I have found 6 problems in current master with TCK 1.1 (104 failed

tests).

Of course, we should run this TCK on CI to be absolutely sure we didn't
miss something.

So the first step is an adding TCK 1.1 suite to our Team City.
I have created sub-task [1] for it and prepared the PR [2].
I need someone with access to merge PR and add suite to Team City.
It going to be just a clone of the current jcache-tck suite, but with

using

the new profile.
You can test new profile locally with the following command:

mvn test -P-release,jcache-tck-1.1 -pl :ignite-core -am


After that, I will start to add sub-task for every problem we have.

Nikolay, can you please help me with merging [1] and adding to the suite?

[1]JIRA: https://issues.apache.org/jira/browse/IGNITE-8687
[2]PR: https://github.com/apache/ignite/pull/4114/files
[3]CI:
https://ci.ignite.apache.org/viewType.html?buildTypeId=
IgniteTests24Java8_RunAll&branch_IgniteTests24Java8=pull/4114/head&tab=
buildTypeStatusDiv

2018-05-23 14:31 GMT+03:00 Александр Меньшиков :


Thanks, Slava. You are right.

2018-05-23 14:00 GMT+03:00 Vyacheslav Daradur :


Hi, Alex!

Please have a look at maven profile named "jcache-tck".

I believe this is what you are looking for.

On Wed, May 23, 2018 at 11:50 AM, Александр Меньшиков
 wrote:

Igniters,
I think I can do it. As I see we already have JCache TCK tests in

TC.

Can I take somewhere settings/script which we are using?

2018-05-23 2:58 GMT+03:00 Dmitriy Setrakyan 
:

Igniters,

It will be great if someone in the community would pick this up.

The

amount

of changes are minimal and many of them only have to do with

clarifying the

documentation. However, removing JSR 107 license confusion in 1.1

would be

great for Ignite.

D.

On Tue, May 22, 2018 at 3:04 PM, Denis Magda 

wrote:

Here is a list of all changes:
https://groups.google.com/forum/#!topic/jsr107/BC1qKqknzKU

The primary argument for the migration is a license. JCache 1.0

is

licensed

by Oracle that causes legal issues for some of the users. Once we

upgrade

to JCache 1.1 the won't longer be a big deal.

However, once we move to 1.1 we need to be sure that we comply

with

the

updated specification.

--
Denis

On Tue, May 22, 2018 at 5:20 AM, Dmitry Pavlov <

dpavlov@gmail.com>

wrote:


Hi Denis,

What was improved in JCache 1.1?

Would it be useful for product to change supported spec.

version?

Sincerely,
Dmitriy Pavlov

пн, 21 мая 2018 г. в 20:12, Denis Magda :


Igniters,

Eventually, JCache was relicensed to Apache 2.0 and released

1.1

version:

https://groups.google.com/forum/#!topic/jsr107/BC1qKqknzKU

Is there anyone interested in upgrading Ignite to the new

version for

the

next release?
https://issues.apache.org/jira/browse/IGNITE-8548

--
Denis




--
Best Regards, Vyacheslav D.

Re: [IGNITE-5714] Design proposal of suspend/resume for pessimistic tx

[Кузнецов Алексей Львович]

PR for this issue is ready to be reviewed.
Changes are relatively small.

1. No network messages have been changed.
2. Thread id usage replaced with xid usage when we work with candidates.
3. All current tests are passed [3]

Please, review. [1], [2]

[1] https://issues.apache.org/jira/browse/IGNITE-5714
[2] https://github.com/apache/ignite/pull/2789
[3] 
https://ci.ignite.apache.org/viewLog.html?buildId=1229177&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_RunAll


28.02.2018 19:10, Aleksey Kuznetsov пишет:
> What if originating(or primary, or couple of primaries) node(s) 
fails when tx is suspended?


Thanks for advice, I've added a couple of failover tests. With the 
following scenarios :


Started transaction, perform some put, suspend it and primary node 
fails. Works as expected : tx rollbacks.
In another scenario, the originating node fails. Works as expected : 
tx rollbacks.


> Do we have tests for this feature?

yeah,we do.

ср, 28 февр. 2018 г. в 14:45, Nikolay Izhikov >:


Hello, Aleksey.

> Tests were added for suspending/resuming optimistic transaction
earlier for parent task.
> I adopted them for pessimistic mode. Also tx rollback tests were
aded.

I think, at least, we need new failover tests.

What if originating(or primary, or couple of primaries) node(s)
fails when tx is suspended?

> Note, that if user suspends transaction and forgets to resume it,
> transaction would be rolled back once timeout has occurred.

Do we have tests for this feature?

В Ср, 28/02/2018 в 10:29 +, Aleksey Kuznetsov пишет:
> Thanx for the answer!
>
> > Can we remove thread id completely from code?
> > Can you estimate how much effort do we need?
>
> I haven't removed thread id completely from the code because we
tightly
> coupled to it in some parts.
> For instance, look at the following scenario :
>
> "When we suspend transaction, we must make sure only thread started
> transaction can suspend it. So we compare thread id in
> `GridNearTxLocal#suspend()`."
>
> Note that cache locks also make use of thread id, consider the
following
> scenario:
>
> "We create cache lock and perform put operation in the same
thread, which
> create implicit transaction. Transaction would compare both
therad ids in
> tx and cache lock and can apply the optimisation: tx would take
advantage
> of cache lock, and would not create its own lock on key."
>
> If we really needed to remove it completely from the code,
significant
> changes are required both to transactions and cache locks.
>
> It could take several weeks to completely remove thread id.
>
> > As far as I can see from parent task [1] we need some complex
tests to be
>
> implemented.
> > Are they presented in prototype?
>
> Tests were added for suspending/resuming optimistic transaction
earlier for
> parent task.
> I adopted them for pessimistic mode.
> Also tx rollback tests were aded.  They check whether timeout
metrics are
> correctly calculated for suspended transactions. You can find
them in
> `GridCacheTransactionalAbstractMetricsSelfTest`.
>
> Please, feel free to propose additional tests for the ticket.
>
>
> вт, 27 февр. 2018 г. в 17:27, Nikolay Izhikov
mailto:nizhi...@apache.org>>:
>
> > Hello, Alexey.
> >
> > Great mail, by the way.
> > I think, it would be great to have this feature in Ignite.
> >
> > > I haven't removed thread id completely from code.
> >
> > Can we remove thread id completely from code?
> > Can you estimate how much effort do we need?
> >
> > As far as I can see from parent task [1] we need some complex
tests to be
> > implemented.
> > Are they presented in prototype?
> >
> > [1]
> >

https://issues.apache.org/jira/browse/IGNITE-4887?focusedCommentId=16069655&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16069655
> >
> > В Пн, 26/02/2018 в 10:59 +, Aleksey Kuznetsov пишет:
> > > Hello Igniters!
> > >
> > > Currently we have suspension/resuming implemented for optimistic
> > > transactions [1].
> > > Unless suspend/resume isn't supported for pessimistic tx JTA
isn't fully
> > > supported [4].
> > >
> > > I’m working on a ticket "Suspension/resuming for pessimistic
> >
> > transactions"
> > > [2].
> > > Goal of the ticket is to support transaction suspend/resume for
> >
> > pessimistic
> > > transactions.
> > >
> > > # Benefits of suspending/resuming transaction.
> > >
> > >   1. Full JTA standart support.
> > >   2. Increase of throughput in high-load scenarios.
> > >       Suspend operation would allow to release Ignite
threads and
> > > op

Re[2]: Welcome

[Протченко Алексей]

Hello all

Thank you, Alexey!


>Вторник, 27 марта 2018, 21:57 +06:00 от Alexey Goncharuk 
>:
>
>Hello Alexey,
>
>Welcome to the Ignite community! I've added you to the contributors list,
>you should be now able to assign the JIRA ticket.
>
>--AG
>
>2018-03-27 18:42 GMT+03:00 Spider (Alexey) < spiderru5...@gmail.com >:
>
>> Hello Ignite Community!
>>
>> My name is Aleksey. I want to contribute to Apache Ignite and want to
>> start with this issue - IGNITE-8049 but I can't assign it to me, my JIRA
>> username astelmak. Any help on this will be appreciated.
>>
>> Thanks!
>>
>>

Hello

[Алексей Рохин]

I would like to join apache ignite development.
My JIRA ID is  rokhin

Going to start from IGNITE-7281.


BR 

iginite participation

[Алексей Рохин]

I would like to join apache ignite development.
My JIRA ID is  r okhin

Going to start from IGNITE-7281.


BR

Grant contributors access

[Алексей Четаев]

Dear Ignite community,

Can you please grant me (lexa) contributor access, so that I can
assign Ignite Jira issues to myself ?

Best Regards
Aleksey Chetaev.