[MTCGA]: new failures in builds [1770510, 1770509] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 09:51:59 MSK 2018 

[MTCGA]: new failures in builds [1770297, 1770296] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 09:07:00 MSK 2018 

[MTCGA]: new failures in builds [1770084] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 08:22:09 MSK 2018 

[MTCGA]: new failures in builds [1770083] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 08:07:07 MSK 2018 

[MTCGA]: new failures in builds [1769864, 1769863] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 07:22:00 MSK 2018 

[MTCGA]: new failures in builds [1769639, 1769638] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 06:54:03 MSK 2018 

[MTCGA]: new failures in builds [1769422, 1769421] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 05:37:03 MSK 2018 

[MTCGA]: new failures in builds [1769208, 1769103] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 04:51:59 MSK 2018 

[MTCGA]: new failures in builds [1769102, 1768890] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 03:53:54 MSK 2018 

[MTCGA]: new failures in builds [1768677] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 03:24:09 MSK 2018 

[MTCGA]: new failures in builds [1768889] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 03:08:56 MSK 2018 

[MTCGA]: new failures in builds [1768676] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 02:39:12 MSK 2018 

[MTCGA]: new failures in builds [1768244, 1768243] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 02:08:53 MSK 2018 

[MTCGA]: new failures in builds [1768886, 1767871, 1767870] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeper&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 Changes may led to failure were done by 
 - andrey.mashenkov 
http://ci.ignite.apache.org/viewModification.html?modId=830304&personal=false

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 Changes may led to failure were done by 
 - andrey.mashenkov 
http://ci.ignite.apache.org/viewModification.html?modId=830304&personal=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 01:22:01 MSK 2018 

Re: GridClosureProcessor.affinityRun() semantics

[David Harvey]

Val,

A fundamental  problem with sending computes to data is that the computes
can be buggy.   We have a set of constantly changing read-only
analytics we want to run and caches where  readFromBackup is set.

If a bug is introduced where the closure goes into an infinite loop, or
consumes all the memory on all nodes it is sent to, we would like that to
be constrained so that if we lose all those nodes, we still have a full
copy of the data on the remaining. We have implemented the simple
affinityBackupFunction which forces the primary and backup to different
sets of nodes (availability zones) on a partition basis, so that no
partition has all of its replicas in the same group of nodes.   If I
use IgniteCompute.Broadcast(), then the execution will be localized to the
cluster group.

However, if I have use cases that want to sent many closures to execute
near a local copy of the data, I'd like to constraint them in the same
way. I can use the Affinity interface to determine the node that
currently has the key, and send a closure there, but the semantics of
affinityRun.. are what I really would like."The data of the partition
where affKey is stored will not be migrated from the target node while the
job is executed."

The semantics of this case should be clear, and they are not.   The code
should be changed to alter the behavior if the primary node is not in the
subgrid.  It should not depend on lower layers detecting and handling a
case it should not have been allowed through.  If the primary and backups
are not in the subgrid, then throw an  exception. I would not consider
the case of code that depends on the current behavior important.

The interesting question is about cost/benefit of defining this as "use a
backup in the subgrid if the the primary is not in the subgrid". The
primary question on cost is if we did choose a backup node because the
primary was not in the grid, would it just work, or would there be ripple
effects at lower layers?

My hope is that we can do a change of modest benefit with low cost, which
could end up being to change the documentation to say not to do this

Thanks,
-DH





On Fri, Aug 31, 2018 at 2:15 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Dave,
>
> In case it's executed even if primary node is outside of the cluster group,
> then I think it's a bug - I would throw an exception in this case. However,
> is there any particular reason you're doing this? Is there a use case? I
> don't see much sense in combining affinityRun with a cluster group.
>
> Backup no node is never used by affinityRun to my knowledge.
>
> -Val
>
> On Fri, Aug 31, 2018 at 7:07 AM David Harvey  wrote:
>
> > This function takes:
> >
> > int partId,
> >
> > ...
> >
> > @Nullable Collection nodes,
> >
> >
> > It uses partId to find the node with the primary partition, and proceeds
> > even if that node is not in the subgrid that was passed in.  This is
> either
> > a bug, or the semantics should be specified more clearly.
> >
> >
> > There are two sub-cases.
> >
> >
> >- one of nodes in the sub-grid is a backup for the partition
> >- the partition does not exist on any of the nodes in the sub-grid
> >
> > This case can be  exposed via IgnuteCompute.affinityRun... when the
> > IgniteCompute was created with a subgrid that did not include the primary
> > node.
> >
> > I got lost tracing the code below this, and could not tell if this would
> > throw an exception or execute on the primary node.   The later would seem
> > to just be a bug.  It would be simple to change this code to choose a
> node
> > in the subgrid or throw and exception.
> >
> > If it selected a backup node, then would the this part of the
> IgniteCompute
> > contract still hold w/o other changes: "The data of the partition where
> > affKey is stored will not be migrated from the target node while the job
> is
> > executed."  ?
> >
> > In any case, the IgniteCompute semantics around this case should be
> stated.
> >
> >
> > -DH
> >
>

[MTCGA]: new failures in builds [1768673, 1767627, 1767625] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeper&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

 *New Critical Failure in master ZooKeeper (Discovery) 2 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery2&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 Changes may led to failure were done by 
 - vpyatkov 
http://ci.ignite.apache.org/viewModification.html?modId=830294&personal=false

 *New Critical Failure in master ZooKeeper (Discovery) 1 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 Changes may led to failure were done by 
 - vpyatkov 
http://ci.ignite.apache.org/viewModification.html?modId=830294&personal=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 00:37:01 MSK 2018 

[MTCGA]: new failures in builds [1768240] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeper&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Sat Sep 01 00:21:50 MSK 2018 

[MTCGA]: new failures in builds [1767867] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeper&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 Changes may led to failure were done by 
 - andrey.mashenkov 
http://ci.ignite.apache.org/viewModification.html?modId=830304&personal=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Fri Aug 31 23:36:50 MSK 2018 

[MTCGA]: new failures in builds [1767623] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New Critical Failure in master ZooKeeper 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeper&branch=%3Cdefault%3E&tab=buildTypeStatusDiv
 Changes may led to failure were done by 
 - vpyatkov 
http://ci.ignite.apache.org/viewModification.html?modId=830294&personal=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Fri Aug 31 22:51:49 MSK 2018 

Re: How can i make my custom plugin serializable - Failed to serialize object: plugins.CustomSecurityProcessor

[Valentin Kulichenko]

There is a non-serializable anonymous class that you have declared within
the plugin. You should check what is actually serialized there and whether
it's supposed to be serialized or not.

-Val

On Fri, Aug 31, 2018 at 10:27 AM wt  wrote:

> How can i make my custom plugin serializable?
>
>
> Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
> authenticate local node (will shutdown local node).
> at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl.localAuthentication(ServerImpl.java:1008)
> at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:879)
> at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
> at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
> at
>
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> ... 10 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> serialize object: plugins.CustomSecurityProcessor$1@350b3a17
> at
>
> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:103)
> at
>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:70)
> at
>
> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:117)
> at
>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
> at
> org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10044)
> at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl.localAuthentication(ServerImpl.java:1002)
> ... 14 more
> Caused by: java.io.NotSerializableException:
> plugins.CustomSecurityProcessor$1
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
>
> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:98)
> ... 19 more
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: GridClosureProcessor.affinityRun() semantics

[Valentin Kulichenko]

Dave,

In case it's executed even if primary node is outside of the cluster group,
then I think it's a bug - I would throw an exception in this case. However,
is there any particular reason you're doing this? Is there a use case? I
don't see much sense in combining affinityRun with a cluster group.

Backup no node is never used by affinityRun to my knowledge.

-Val

On Fri, Aug 31, 2018 at 7:07 AM David Harvey  wrote:

> This function takes:
>
> int partId,
>
> ...
>
> @Nullable Collection nodes,
>
>
> It uses partId to find the node with the primary partition, and proceeds
> even if that node is not in the subgrid that was passed in.  This is either
> a bug, or the semantics should be specified more clearly.
>
>
> There are two sub-cases.
>
>
>- one of nodes in the sub-grid is a backup for the partition
>- the partition does not exist on any of the nodes in the sub-grid
>
> This case can be  exposed via IgnuteCompute.affinityRun... when the
> IgniteCompute was created with a subgrid that did not include the primary
> node.
>
> I got lost tracing the code below this, and could not tell if this would
> throw an exception or execute on the primary node.   The later would seem
> to just be a bug.  It would be simple to change this code to choose a node
> in the subgrid or throw and exception.
>
> If it selected a backup node, then would the this part of the IgniteCompute
> contract still hold w/o other changes: "The data of the partition where
> affKey is stored will not be migrated from the target node while the job is
> executed."  ?
>
> In any case, the IgniteCompute semantics around this case should be stated.
>
>
> -DH
>

[GitHub] ignite pull request #4662: IGNITE-9396 ML Examples: can't run examples, not ...

[oignatenko]

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/4662

IGNITE-9396 ML Examples: can't run examples, not enough dependencies in 
pom.xml



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9396

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4662.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4662


commit 1eeca908a8076a8317947dac8a46845964d1d7ea
Author: Oleg Ignatenko 
Date:   2018-08-23T13:13:28Z

IGNITE-9348 ML examples improvements
- wip (logging improved)
-- verified with diffs overview, executing the examples and studying 
execution logs

commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f
Author: Oleg Ignatenko 
Date:   2018-08-23T14:45:57Z

Revert "IGNITE-9348 ML examples improvements"

This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea.

commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674
Author: Oleg Ignatenko 
Date:   2018-08-23T14:57:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9642b233b5701bdad47ebea163079160227c582a
Author: Oleg Ignatenko 
Date:   2018-08-28T14:01:11Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4
Author: Oleg Ignatenko 
Date:   2018-08-28T15:13:02Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit d2caba67b156674f051f50faebeafe0871bf0914
Author: Oleg Ignatenko 
Date:   2018-08-29T13:14:07Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 16775dff51d71ea68b4a3dea98be552130c493ed
Author: Oleg Ignatenko 
Date:   2018-08-30T09:00:56Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aedb59929974fe205b949225c1a338c68c60cfc8
Author: Oleg Ignatenko 
Date:   2018-08-30T09:42:38Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 39c6482fcdca506aa33011ed21c98060b4a8c68b
Author: Oleg Ignatenko 
Date:   2018-08-30T11:28:05Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 33b32a2760a6559c78283b927e3191180d8ed9e1
Author: Oleg Ignatenko 
Date:   2018-08-30T12:31:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0
Author: Oleg Ignatenko 
Date:   2018-08-30T14:06:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3
Author: Oleg Ignatenko 
Date:   2018-08-30T16:41:51Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aacac88db519187413b0fc5ff9d0e55b8f8cff22
Author: Oleg Ignatenko 
Date:   2018-08-31T10:12:32Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 897f920dde46022849b13f9fb86dba8e54119a56
Author: Oleg Ignatenko 
Date:   2018-08-31T13:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 114c79e54c1b316006ccc3ff22d20d902f9313df
Author: Oleg Ignatenko 
Date:   2018-08-31T17:39:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit edb68a4ae2ccd1e66d77aa3690d4e0c541cf78b4
Author: Oleg Ignatenko 
Date:   2018-08-31T17:56:57Z

IGNITE-9396 ML Examples: can't run examples, not enough dependencies in 
pom.xml
- documentation added as per discussion in JIRA ticket comments
-- verified with diffs overview and clean build of examples per added 
instructions




---

How can i make my custom plugin serializable - Failed to serialize object: plugins.CustomSecurityProcessor

[wt]

How can i make my custom plugin serializable?


Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
authenticate local node (will shutdown local node).
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.localAuthentication(ServerImpl.java:1008)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:879)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 10 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
serialize object: plugins.CustomSecurityProcessor$1@350b3a17
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:103)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:70)
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:117)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10044)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.localAuthentication(ServerImpl.java:1002)
... 14 more
Caused by: java.io.NotSerializableException:
plugins.CustomSecurityProcessor$1
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:98)
... 19 more



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #4623: IGNITE-9373: Fix mvcc tests.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4623


---

[GitHub] ignite pull request #4661: IGNITE-9448

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4661


---

[GitHub] ignite pull request #4661: IGNITE-9448

[vldpyatkov]

GitHub user vldpyatkov opened a pull request:

https://github.com/apache/ignite/pull/4661

IGNITE-9448

Change ZooKeeper version to 3.4.13

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9448

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4661.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4661


commit 33d7c41b45b422845314e0c2ccebfd36a706d243
Author: vd-pyatkov 
Date:   2018-08-31T15:39:58Z

IGNITE-9448
Change ZooKeeper version to 3.4.13




---

Re: Backup queue for local continuous query

[Andrey Mashenkov]

Sorry for late answer.

Yes, I'm agree local CQ should ignore backup partitions at all for
partitioned caches,
and we should trigger all events (primary and backups) for replicated
caches.

With this approach there is no need to collect backup events in queue
anymore.

On Fri, Aug 10, 2018 at 4:51 PM Denis Mekhanikov 
wrote:

> Andrey,
>
> I also think, that remote nodes shouldn't be involved.
>
> > Instead of using backup queue we can notify CQ on backup events instantly
> This would be an incompatible change. It would break users' code, that
> doesn't expect notifications on backup entries.
> On the other hand, it would make our API more consistent, since CQ events
> are triggered for all entries, if a cache is replicated.
> We could add *CacheEntryEvent#isPrimary* property, that would let users
> distinguish primary and backup events.
>
> But since it would change behaviour, I'm inclined to just ignore events on
> backup partitions.
>
> What do you think?
>
> Denis
>
> пт, 10 авг. 2018 г. в 13:41, Andrey Mashenkov  >:
>
> > Hi Denis,
> >
> > As my understanding right, to prevent potential OOM Local CQ either
> > shouldn't collect anything in backup queue or other nodes should send
> akcs.
> >
> > Second way look strange as Local CQ runs on local node only and doesn't
> > care whats going on other nodes.
> > It is not clear what events and when other nodes should ack as they
> doesn't
> > participate in query?
> > When a primary node for a partition goes away and node with local CQ
> > running become a new primary
> > then seems, it make no sense to notify CQ with stale events from backup
> > queue and only new events make sense (as from new primary).
> >
> > Instead of using backup queue we can notify CQ on backup events instantly
> > and let user filter out backup events if needed in listener.
> >
> > So, I'd expect local query shouldn't involve remote nodes at all.
> > Also, I'm not sure we should filter out backup events. Can we let users
> to
> > do this in their listener code as it shouldn't costs for local CQ?
> >
> >
> >
> >
> >
> > On Thu, Aug 2, 2018 at 3:35 PM Denis Mekhanikov 
> > wrote:
> >
> > > Igniters,
> > >
> > > I noticed, that local continuous queries store queues with backup
> > entries.
> > > And since the queries are local, other nodes never send acknowledges.
> > >
> > > It has two negative effects:
> > > 1) Backup queue may grow indefinitely and cause OOME.
> > > 2) When topology changes, the backup queue is flushed, and old
> > > notifications are delivered to the listener, even if they happened long
> > > time ago.
> > >
> > > I prepared a fix for it. You can find it in the ticket: IGNITE-9009
> > > 
> > > It makes local continuous queries ignore backup entries and not put
> them
> > > into the backup queues.
> > >
> > > Nikolay, Semyon,
> > > You are the people, who implemented this piece of functionality.
> > > Could you review these changes and share your opinion?
> > >
> > > I can think of another way to fix it: we could register
> > > *CacheContinuousQueryHandlers* on all nodes even for a local CQ, and
> make
> > > them send backup notifications to the subscriber node.
> > > It will help in cases, when you create local listeners on all nodes and
> > > want all nodes to process changes, that happen on their primary
> > partitions.
> > > In this case backup entries may help not to lose updates. But actually
> it
> > > will leave a time window between sending a backup acknowledge and
> > notifying
> > > the local listener. And such use-case will generate quadratic number of
> > > handlers and pollute the network communication.
> > >
> > > So, I prefer the first option.
> > >
> > > What do you think?
> > >
> > > Denis
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>


-- 
Best regards,
Andrey V. Mashenkov

Re: Compression prototype

[Ilya Kasnacheev]

Just as I have started praising Zstd, it began to show JVM crashes in
native code in train dict :(

I guess it has limits to train buffer, after which errorneous behaviour is
exhibited. Maybe we will need to submit a pull request:)

Regards,
-- 
Ilya Kasnacheev


пт, 31 авг. 2018 г. в 11:56, Ilya Kasnacheev :

> Hello!
>
> I am testing Zstd with dictionary, and it looks very very promising. I'm
> confident I can choose settings where it is faster than my own algo while
> bringing better compression ratio, on "cod" dataset.
>
> So I am happliy retiring my code and switching to Zstd. Would probably
> mean that we will ship compression implementation as a separate module.
>
> It is a pity that I did not find out about Zstd dictionary support
> earlier, that would mean I could skip a few days of work.
>
> Without dictionary the results of Zstd were worse than my own algo, but it
> was faster.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 27 авг. 2018 г. в 14:53, Vyacheslav Daradur :
>
>> According to my benchmarks - zstd compression algorithm [1] looks very
>> interesting, it has a high compression ratio with quite good speed.
>> AFAIK it supports external dictionaries, but I'm not sure about using
>> it with "on the fly building" dictionaries. Anyway, have look at (it
>> has ASF 2.0 friendly license).
>>
>> Also, here is data generator / loader [1]. If it will be useful for
>> you we should ask Nikolay Izhikov to share public docs to start.
>>
>> [1] https://github.com/facebook/zstd
>> [2] https://github.com/nizhikov/ignite-cod-data-loader
>> On Mon, Aug 27, 2018 at 2:11 PM Ilya Kasnacheev
>>  wrote:
>> >
>> > Hello Vyacheslav!
>> >
>> > Unfortunately I have not found any efficient algorithms that will allow
>> me
>> > to use external dictionary as a pre-processed data structure. If plain
>> gzip
>> > is used without dictionary, the compression is around 0.7, as opposed to
>> > 0.4 that I will get with custom implementation, AFAIR the performance
>> was
>> > also worse. I didn't really try it with dictionary, but I assume
>> > performance will be even worse since it will have to scan dictionary
>> before
>> > getting to actual data.
>> >
>> > We have such a huge array of tests that we can just run them all with
>> > compression enabled, see if there are any new failures. But the impact
>> of
>> > my commit is fairly low, it is only triggered when data is written to
>> page
>> > (maybe to WAL also?), and we don't really do much frivolous stuff to
>> pages.
>> >
>> > Still, I am very much interested in finding existing compression
>> > implementations with support of external dictionary; I am also very much
>> > interested in having different implementations of compression for Apache
>> > Ignite (such as per page compression) and comparing them by benchmark
>> and
>> > by code impact. I am also very interested in large standard datasets for
>> > Apache Ignite (or generators thereof) so that we can run precise
>> benchmarks
>> > on various compression schemes. If you have any of the following, please
>> > get back to me.
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > пн, 27 авг. 2018 г. в 11:35, Vyacheslav Daradur :
>> >
>> > > Hi Igniters!
>> > >
>> > > Ilya, I'm glad to see one more person who is interested in the
>> > > compression feature in Ignite.
>> > >
>> > > I looked through the pull request and want to share following
>> thoughts:
>> > >
>> > > It's very dangerous using a custom algorithm in this way - you store
>> > > serialized data separate from a dictionary and there are a lot of
>> > > points when we may lose data: rebalancing, serialization errors, node
>> > > rebooting and so on.
>> > >
>> > > I'd suggest the following ways to improve reliability:
>> > > - use well know algorithms: zstd, deflater, lzma, gzip e.g. that
>> > > allows us to decompress data in any situation
>> > > - store the dictionary inside page with data
>> > >
>> > > Also, we have a lot of discussions [1] [2] about compression on
>> > > BinaryObject and BinaryMarshaller level and Vladimir Ozerov was
>> > > strictly against a compression on this level.
>> > > If something has changed since then, you may look through [1] [2] [3]
>> > > I've done a lot of research in algorithms comparison it may be useful
>> > > for you.
>> > >
>> > > [1]
>> > >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html
>> > > [2]
>> > >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-td20679.html
>> > > [3] https://issues.apache.org/jira/browse/IGNITE-3592
>> > > [4] https://issues.apache.org/jira/browse/IGNITE-5226
>> > > [5] https://github.com/daradurvs/ignite-compression
>> > > On Sat, Aug 25, 2018 at 2:51 AM Denis Magda 
>> wrote:
>> > > >
>> > > > >
>> > > > > Currently, the dictionary for decompression is only stored on
>> heap.
>> > > After
>> > > > > restart there's compressed data in the PDS, but there's no
>> dictionary
>> > > :)
>> > > >
>> > 

[jira] [Created] (IGNITE-9448) Change ZooKeeper version to 3.4.13

[Vladislav Pyatkov (JIRA)]

Vladislav Pyatkov created IGNITE-9448:
-

 Summary: Change ZooKeeper version to 3.4.13
 Key: IGNITE-9448
 URL: https://issues.apache.org/jira/browse/IGNITE-9448
 Project: Ignite
  Issue Type: Test
  Components: zookeeper
Reporter: Vladislav Pyatkov


Should to change ZooKeeper dependency to last release - just now it is 3.4.13.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9447) Benchmarks hangs intemittently due to distributed race condition.

[Pavel Kuznetsov (JIRA)]

Pavel Kuznetsov created IGNITE-9447:
---

 Summary: Benchmarks hangs intemittently due to distributed race 
condition.
 Key: IGNITE-9447
 URL: https://issues.apache.org/jira/browse/IGNITE-9447
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Pavel Kuznetsov
Assignee: Pavel Kuznetsov


If we run more than one yardstick driver, benchmark hangs intermittently.

We've got yardstick's base driver class 
org.apache.ignite.yardstick.IgniteAbstractBenchmark it has logic to wait all 
the nodes in the cluster.

{noformat}
final CountDownLatch nodesStartedLatch = new CountDownLatch(1);

ignite().events().localListen(new IgnitePredicate() {
@Override public boolean apply(Event gridEvt) {
if (nodesStarted())
nodesStartedLatch.countDown();

return true;
}
}, EVT_NODE_JOINED);

if (!nodesStarted()) {
println(cfg, "Waiting for " + (args.nodes() - 1) + " nodes to 
start...");

nodesStartedLatch.await();
}
{noformat}

This code is executed on every driver node.
If we want to close local ignite instance just after cluster is ready 
(containing expected number of nodes), sometimes we'll have dead lock:

1) cluster contains N-1 nodes, all nodes are waiting for the Nth node.
2) Nth node is connected, cluster receives message, waitForNodes code of Nth 
node is not executed.
3) N-1 nodes got this message and stop waiting.
4) N-1 thinks that cluster is ready and call ignite.close() on their local 
instances
5) Nth node starts waiting for cluster to contain number of nodes, but N-1 of 
them closed their instances
6) Nth node is waiting infinitely. 

We can avoid this problem if we use distributed CountDownLatch



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4660: IGNITE-9320: Mvcc cache configuration and cluster...

[rkondakov]

GitHub user rkondakov opened a pull request:

https://github.com/apache/ignite/pull/4660

IGNITE-9320: Mvcc cache configuration and cluster nodes validation.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9320

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4660.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4660


commit b00fd2a7d315e194b630f4ac2ba09842e69de48d
Author: rkondakov 
Date:   2018-08-30T09:09:59Z

IGNITE-9320: Mvcc configuration changed from global flag to per cache flag. 
Tests fixed.

commit 1305e54d0afa046aa1100824b9c9a56bc5f84d7a
Author: rkondakov 
Date:   2018-08-31T11:20:15Z

Start mvcc processor and cluster validation WIP




---

[GitHub] ignite pull request #4537: IGNITE-8189

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4537


---

GridClosureProcessor.affinityRun() semantics

[David Harvey]

This function takes:

int partId,

...

@Nullable Collection nodes,


It uses partId to find the node with the primary partition, and proceeds
even if that node is not in the subgrid that was passed in.  This is either
a bug, or the semantics should be specified more clearly.


There are two sub-cases.


   - one of nodes in the sub-grid is a backup for the partition
   - the partition does not exist on any of the nodes in the sub-grid

This case can be  exposed via IgnuteCompute.affinityRun... when the
IgniteCompute was created with a subgrid that did not include the primary
node.

I got lost tracing the code below this, and could not tell if this would
throw an exception or execute on the primary node.   The later would seem
to just be a bug.  It would be simple to change this code to choose a node
in the subgrid or throw and exception.

If it selected a backup node, then would the this part of the IgniteCompute
contract still hold w/o other changes: "The data of the partition where
affKey is stored will not be migrated from the target node while the job is
executed."  ?

In any case, the IgniteCompute semantics around this case should be stated.


-DH

[jira] [Created] (IGNITE-9446) MVCC: Races in Vacuum worker on cache stop.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-9446:


 Summary: MVCC: Races in Vacuum worker on cache stop.
 Key: IGNITE-9446
 URL: https://issues.apache.org/jira/browse/IGNITE-9446
 Project: Ignite
  Issue Type: Bug
Reporter: Andrew Mashenkov


For now, partition store BTree can be destroyed while vacuum in progress 
regardless partition reservations which causes VacuumTask failure.

We should fix vacuum failure properly in that case.

This should fix failed TC test 
CacheMvccPartitionedSqlQueriesTest.testDistributedJoinSimple()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Looking for information on these methods

[Ilya Kasnacheev]

Hello!

authenticate() is called from Client and REST initiation code.
authenticatedSubject()
is called on node start. authorize() is called from all over the code base,
when need to check if permission is granted.

My recommendation is to check out Apache Ignite source, get accustomized
with navigating it. Any recurring questions you should direct to developer
list since plugin development falls in that area IMHO.

Regards,
-- 
Ilya Kasnacheev


пт, 31 авг. 2018 г. в 14:47, wt :

> okay i have a white list plugin class that is now working - ThankYou Ilya
> (GridGain)
>
> I can now control nodes joining the cluster be it clients or servers. My
> next task is to manage user requests to specific caches and what they can
> do
> on those. I already have the GridSecurityProcessor class as part of my
> implementation and the class has the following methods
>
> SecurityContext authenticate(AuthenticationContext var1)
>
> SecuritySubject authenticatedSubject(UUID var1)
>
> void authorize(String var1, SecurityPermission var2, @Nullable
> SecurityContext var3)
>
>
> my questions are:
>
> 1) do these methods get called when requests come in from nodes and
> clients
> and are they used anywhere else
>
> 2) i have the GridSecurityProcessor  working already for node
> authentication, is there anything that is needed besides implementing the
> logic on these methods. For this i am focusing on the server side only and
> will implement the client side logic on the jdbc and odbc later.
>
> 3) if i am wrong in my assumption on request authentication and
> authorization in context to these methods, what classes\methods should i
> be
> looking at.
>
>
> As always, thank you in advanced (i couldn't do this without the support
> of
> this awesome community)
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[GitHub] ignite pull request #4659: IGNITE-9387: update interface for trainers

[avplatonov]

GitHub user avplatonov opened a pull request:

https://github.com/apache/ignite/pull/4659

IGNITE-9387: update interface for trainers



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9387

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4659.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4659


commit 7633191513acacf689002584a7cbaf77e88beb13
Author: Alexey Platonov 
Date:   2018-08-27T11:18:17Z

initial commit

commit 3f2d5a4fd1c40ac243dcef30e062c69425ef1dec
Author: Alexey Platonov 
Date:   2018-08-28T13:12:56Z

update DatasetTrainer interface

commit 8be18d70db897583c031cb890a1f5e204e909752
Author: Alexey Platonov 
Date:   2018-08-28T15:01:09Z

 SVM update interface implementation

commit 783a14efed7a526dcad667475d1b3ace4817321b
Author: Alexey Platonov 
Date:   2018-08-28T15:34:59Z

MLP update interface realization

commit 3b4eb27f049031f41985e9cafdebf6f6b660a9fe
Author: Alexey Platonov 
Date:   2018-08-29T09:00:32Z

LSQR update interface impl

commit 6ebe65517b29e8a5cbd248f9fcf88bc5f5751073
Author: Alexey Platonov 
Date:   2018-08-29T09:47:11Z

Logistic regression model update interface impl

commit 8a65426175283234df2b40379a4c7838a665ca38
Author: Alexey Platonov 
Date:   2018-08-29T09:57:44Z

Linear regression update interface impl

commit 346fff9117ad039e9ae6b54d783f8b3d442ba461
Author: Alexey Platonov 
Date:   2018-08-29T11:22:48Z

fix performance degradation due to Optional.orElse

commit 1bff50603daa762213ff85f035c4c788e3206229
Author: Alexey Platonov 
Date:   2018-08-29T11:46:11Z

kmeans update iface impl

commit 1c509094330870dc6e7de9a1b6054218d02b51c5
Author: Alexey Platonov 
Date:   2018-08-29T12:30:58Z

KNN update iface impl

commit a360ad417d8c9a9a0d367e216a744e2c345335ce
Author: Alexey Platonov 
Date:   2018-08-29T13:05:25Z

Bagging model update iface impl

commit cd66719ad0f1c8e537905ac3197d0b509e139fe7
Author: Alexey Platonov 
Date:   2018-08-29T15:58:29Z

ANN update iface impl

commit 267ab3f09266a97520eb3c8f1a023f310d1b564c
Author: Alexey Platonov 
Date:   2018-08-30T08:33:26Z

DT update iface support

commit 96411664f1701075d9074f51bdd994ddf527fb67
Author: Alexey Platonov 
Date:   2018-08-30T08:54:49Z

little refactoring

commit c9d1563c4691a72ac1e1d7722db29609243ea511
Author: Alexey Platonov 
Date:   2018-08-30T09:15:59Z

Impl of Exportable iface for ModelsComposition

commit e5f2c63a2f9b7d30aeb5c5a169e52042f37279ce
Author: Alexey Platonov 
Date:   2018-08-30T10:03:33Z

add checkState iface

commit 5bbac78355b7bc7ece578cac9ffb87d1bd7b2e0f
Author: Alexey Platonov 
Date:   2018-08-30T10:26:22Z

implement simple checkState for models

commit c2e3dfec9b5dd4968cba5f678d09a265a3d5183c
Author: Alexey Platonov 
Date:   2018-08-30T12:33:30Z

add simple test to update for KMeans and KNN/ANN

commit 9e740f590316dea69091132a11dad9c168a65be5
Author: Alexey Platonov 
Date:   2018-08-31T09:03:37Z

merge with master

commit 17070a8738c594fdf3616b2101342d62d94f291c
Author: Alexey Platonov 
Date:   2018-08-31T09:04:12Z

merge with master

commit 635be783c1622657d68cfdcb46a23c917db3678e
Author: Alexey Platonov 
Date:   2018-08-31T12:09:22Z

add tests for corner cases for updating and shade NPE when dataset is empty 
while first learning

commit 669e5adf70c75e6e8eefd74f4f158292eec81c22
Author: Alexey Platonov 
Date:   2018-08-31T13:04:05Z

pre review check

commit 9299c47bb71196daca06fe3fec703c62eef865f1
Author: Alexey Platonov 
Date:   2018-08-31T13:13:36Z

pre review check




---

[jira] [Created] (IGNITE-9445) Use valid tag for page write unlock while reading cold page from disk.

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-9445:
-

 Summary: Use valid tag for page write unlock while reading cold 
page from disk.
 Key: IGNITE-9445
 URL: https://issues.apache.org/jira/browse/IGNITE-9445
 Project: Ignite
  Issue Type: Bug
Reporter: Alexei Scherbakov
Assignee: Alexei Scherbakov


The problem arises when passing pageId with not actual page rotation tag to 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl#acquirePage(int,
 long, boolean).

It's not possible in advance to know the actual value without reading stored 
page.

Such scenario may lead to locked forever page if passed and persisted tags are 
different.

Solution - unlock page using actual(persisted) tag value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4658: IGNITE-9438 fix standaloneWalRecordsIterator file...

[antonovsergey93]

GitHub user antonovsergey93 opened a pull request:

https://github.com/apache/ignite/pull/4658

IGNITE-9438 fix standaloneWalRecordsIterator file descriptors leak.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite IGNITE-9438

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4658.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4658


commit 0f61bec40c021e3ac0aa0a62d401d6b0cbaed819
Author: Sergey Antonov 
Date:   2018-08-31T12:01:21Z

IGNITE-9438 fix standaloneWalRecordsIterator file descriptors leak.




---

Looking for information on these methods

[wt]

okay i have a white list plugin class that is now working - ThankYou Ilya 
(GridGain) 

I can now control nodes joining the cluster be it clients or servers. My 
next task is to manage user requests to specific caches and what they can do 
on those. I already have the GridSecurityProcessor class as part of my 
implementation and the class has the following methods 

SecurityContext authenticate(AuthenticationContext var1) 

SecuritySubject authenticatedSubject(UUID var1) 

void authorize(String var1, SecurityPermission var2, @Nullable 
SecurityContext var3) 


my questions are: 

1) do these methods get called when requests come in from nodes and clients 
and are they used anywhere else 

2) i have the GridSecurityProcessor  working already for node 
authentication, is there anything that is needed besides implementing the 
logic on these methods. For this i am focusing on the server side only and 
will implement the client side logic on the jdbc and odbc later. 

3) if i am wrong in my assumption on request authentication and 
authorization in context to these methods, what classes\methods should i be 
looking at. 


As always, thank you in advanced (i couldn't do this without the support of 
this awesome community) 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-9444) TcpDiscoveryCloudIpFinderSelfTest.testRackspace fails in master

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-9444:


 Summary: TcpDiscoveryCloudIpFinderSelfTest.testRackspace fails in 
master
 Key: IGNITE-9444
 URL: https://issues.apache.org/jira/browse/IGNITE-9444
 Project: Ignite
  Issue Type: Test
Reporter: Alexey Goncharuk






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4645: IGNITE-9424 set partition to KeyCacheObject after...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4645


---

[jira] [Created] (IGNITE-9443) IgniteSet iterator created on the client node is not working with the replicated cache.

[Pavel Pereslegin (JIRA)]

Pavel Pereslegin created IGNITE-9443:


 Summary: IgniteSet iterator created on the client node is not 
working with the replicated cache.
 Key: IGNITE-9443
 URL: https://issues.apache.org/jira/browse/IGNITE-9443
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Affects Versions: 2.6
Reporter: Pavel Pereslegin
Assignee: Pavel Pereslegin


IgniteSet iterator created on the client node is not working with the 
replicated cache.

Reproducer:
{code:java}
public void testClientIteratorOnReplicatedCache() throws Exception {
startGrid(0);

Ignition.setClientMode(true);

Ignite client = startGrid(1);

Ignition.setClientMode(false);

IgniteSet set = client.set("test", new 
CollectionConfiguration().setCacheMode(CacheMode.REPLICATED));

set.add(1);

assertTrue(set.iterator().hasNext());
}
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9442) Collocated IgniteSet#close is not working on non-affinity node.

[Pavel Pereslegin (JIRA)]

Pavel Pereslegin created IGNITE-9442:


 Summary: Collocated IgniteSet#close is not working on non-affinity 
node.
 Key: IGNITE-9442
 URL: https://issues.apache.org/jira/browse/IGNITE-9442
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Affects Versions: 2.6
Reporter: Pavel Pereslegin
Assignee: Pavel Pereslegin


Simple reproducer:

{code:java}
public void testSetBlockOnCloseFromNonAffinityNode() throws Exception {
Ignite node = startGridsMultiThreaded(2);

ClusterNode locNode = node.cluster().localNode();

IgniteSet set = node.set("test",
new CollectionConfiguration().setCollocated(true).setNodeFilter(n 
-> !F.eqNodes(locNode, n)));

set.close();

GridTestUtils.assertThrows(log, new Callable() {
@Override public Void call() throws Exception {
set.add(1);

return null;
}
}, IllegalStateException.class, null);
}
{code}

IgniteSet should be blocked on all nodes before cleanup data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Compression prototype

[Ilya Kasnacheev]

Hello!

I am testing Zstd with dictionary, and it looks very very promising. I'm
confident I can choose settings where it is faster than my own algo while
bringing better compression ratio, on "cod" dataset.

So I am happliy retiring my code and switching to Zstd. Would probably mean
that we will ship compression implementation as a separate module.

It is a pity that I did not find out about Zstd dictionary support earlier,
that would mean I could skip a few days of work.

Without dictionary the results of Zstd were worse than my own algo, but it
was faster.

Regards,
-- 
Ilya Kasnacheev


пн, 27 авг. 2018 г. в 14:53, Vyacheslav Daradur :

> According to my benchmarks - zstd compression algorithm [1] looks very
> interesting, it has a high compression ratio with quite good speed.
> AFAIK it supports external dictionaries, but I'm not sure about using
> it with "on the fly building" dictionaries. Anyway, have look at (it
> has ASF 2.0 friendly license).
>
> Also, here is data generator / loader [1]. If it will be useful for
> you we should ask Nikolay Izhikov to share public docs to start.
>
> [1] https://github.com/facebook/zstd
> [2] https://github.com/nizhikov/ignite-cod-data-loader
> On Mon, Aug 27, 2018 at 2:11 PM Ilya Kasnacheev
>  wrote:
> >
> > Hello Vyacheslav!
> >
> > Unfortunately I have not found any efficient algorithms that will allow
> me
> > to use external dictionary as a pre-processed data structure. If plain
> gzip
> > is used without dictionary, the compression is around 0.7, as opposed to
> > 0.4 that I will get with custom implementation, AFAIR the performance was
> > also worse. I didn't really try it with dictionary, but I assume
> > performance will be even worse since it will have to scan dictionary
> before
> > getting to actual data.
> >
> > We have such a huge array of tests that we can just run them all with
> > compression enabled, see if there are any new failures. But the impact of
> > my commit is fairly low, it is only triggered when data is written to
> page
> > (maybe to WAL also?), and we don't really do much frivolous stuff to
> pages.
> >
> > Still, I am very much interested in finding existing compression
> > implementations with support of external dictionary; I am also very much
> > interested in having different implementations of compression for Apache
> > Ignite (such as per page compression) and comparing them by benchmark and
> > by code impact. I am also very interested in large standard datasets for
> > Apache Ignite (or generators thereof) so that we can run precise
> benchmarks
> > on various compression schemes. If you have any of the following, please
> > get back to me.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 27 авг. 2018 г. в 11:35, Vyacheslav Daradur :
> >
> > > Hi Igniters!
> > >
> > > Ilya, I'm glad to see one more person who is interested in the
> > > compression feature in Ignite.
> > >
> > > I looked through the pull request and want to share following thoughts:
> > >
> > > It's very dangerous using a custom algorithm in this way - you store
> > > serialized data separate from a dictionary and there are a lot of
> > > points when we may lose data: rebalancing, serialization errors, node
> > > rebooting and so on.
> > >
> > > I'd suggest the following ways to improve reliability:
> > > - use well know algorithms: zstd, deflater, lzma, gzip e.g. that
> > > allows us to decompress data in any situation
> > > - store the dictionary inside page with data
> > >
> > > Also, we have a lot of discussions [1] [2] about compression on
> > > BinaryObject and BinaryMarshaller level and Vladimir Ozerov was
> > > strictly against a compression on this level.
> > > If something has changed since then, you may look through [1] [2] [3]
> > > I've done a lot of research in algorithms comparison it may be useful
> > > for you.
> > >
> > > [1]
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html
> > > [2]
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-td20679.html
> > > [3] https://issues.apache.org/jira/browse/IGNITE-3592
> > > [4] https://issues.apache.org/jira/browse/IGNITE-5226
> > > [5] https://github.com/daradurvs/ignite-compression
> > > On Sat, Aug 25, 2018 at 2:51 AM Denis Magda  wrote:
> > > >
> > > > >
> > > > > Currently, the dictionary for decompression is only stored on heap.
> > > After
> > > > > restart there's compressed data in the PDS, but there's no
> dictionary
> > > :)
> > > >
> > > >
> > > > Basically, it means that I've lost my data, right? How about
> persisting
> > > > data to disk.
> > > >
> > > > Overall, we need Vladimir Ozerov to check the contribution. He was
> the
> > > one
> > > > who sponsored the IEP and knows the area best.
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Fri, Aug 24, 2018 at 4:31 AM Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > It is

Re: [MTCGA]: new failures in builds [1757777] needs to be handled

[Alexey Goncharuk]

The failure is not related to the change (verified in a separate branch
with revert) and seems to be related to some change in rackspace
infrastructure. Will create a ticket and mute the test.

пт, 31 авг. 2018 г. в 0:07, :

> Hi Ignite Developer,
>
> I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> I hope you can help.
>
>  *New test failure in master
> TcpDiscoveryCloudIpFinderSelfTest.testRackspace
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=3102247230760992596&branch=%3Cdefault%3E&tab=testDetails
>  Changes may led to failure were done by
>  - irakov
> http://ci.ignite.apache.org/viewModification.html?modId=829932&personal=false
>
> - If your changes can led to this failure(s), please create issue
> with label MakeTeamCityGreenAgain and assign it to you.
> -- If you have fix, please set ticket to PA state and write to dev
> list fix is ready
> -- For case fix will require some time please mute test and set
> label Muted_Test to issue
> - If you know which change caused failure please contact change
> author directly
> - If you don't know which change caused failure please send
> message to dev list to find out
> Should you have any questions please contact dev@ignite.apache.org
> Best Regards,
> MTCGA.Bot
> Notification generated at Fri Aug 31 00:06:59 MSK 2018
>

[GitHub] ignite pull request #4641: IGNITE-9348 ML examples improvements

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4641


---

Re: Hello ignite team, IGNITE-9433 contribution

[Alexey Goncharuk]

Hi Pavel,

Welcome to the Ignite community! I've added you to the contributors list,
you should now be able to assign the tickets to yourself.

--AG

чт, 30 авг. 2018 г. в 16:08, Pavel Voronkin :

> Hi Team,
>
> I'd like to join to Apache Ignite development.
> Especially to the ML part.
> Currently, I work on IGNITE-9433
> https://issues.apache.org/jira/browse/IGNITE-9433
>
> My Jira login is voropava
>
> Best regards,
> Pavel
>

[GitHub] ignite pull request #4657: IGNITE-9441 check crc only if record read correct

[akalash]

GitHub user akalash opened a pull request:

https://github.com/apache/ignite/pull/4657

IGNITE-9441 check crc only if record read correct



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4657.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4657


commit 337bfe6c0b728e94f00c8552ca001a438338e40e
Author: Anton Kalashnikov 
Date:   2018-08-31T07:58:04Z

IGNITE-9441 check crc only if record read correct




---

[jira] [Created] (IGNITE-9441) Failed to read WAL record at position

[Anton Kalashnikov (JIRA)]

Anton Kalashnikov created IGNITE-9441:
-

 Summary: Failed to read WAL record at position
 Key: IGNITE-9441
 URL: https://issues.apache.org/jira/browse/IGNITE-9441
 Project: Ignite
  Issue Type: Test
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov


IgnitePdsAtomicCacheHistoricalRebalancingTest.testPartitionCounterConsistencyOnUnstableTopology
IgnitePdsAtomicCacheHistoricalRebalancingTest.testTopologyChangesWithConstantLoad
IgnitePdsTxHistoricalRebalancingTest.testPartitionCounterConsistencyOnUnstableTopology



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)