Re: Brainstorm: Make TC Run All faster

[Павлухин Иван]

Nikolay,

> I thought by "faster" we all meant time between One push RunAll button and 
> one get results.
> What else definition are possible here?

But this "faster" depends on a current load. E.g. if we have a lot of
idle slaves then incresead parallelism will allow a build to complete
"faster". If we have many slaves busy than parallelism might "slow"
down an execution. I predict that you are mostly interested in average
metrics when TC has a build queue as it usually have on weekdays. If
so then aggregatin other related jobs into "Build Apache Ignite" might
speed execution of RunAll measured in wall clock time.

And also I would like to return to a point I already mentioned earlier
in other conversations. Our CI environment suites perfectly for an
agile way of working. I suppose that we could try different setups
almost freely and see if it gives a desired effect (and rollback if
no). I am ok to go with Vyacheslav proposal. But I would like to
observe some metrics showing that builds become running faster than
before. Can we do this?

вт, 30 апр. 2019 г. в 17:42, Dmitriy Pavlov :
>
> Hi, One more metric we can introduce is efficiency. Test run time/overall
> time. The goal of launching RunAll is to obtain tests results.
>
> I've added more statistics to be displayed in the TC Bot, so now you may
> click 'more' button near 'chain results' caption and you'll see sum of
> statistics for all chain or individual suite. Latest RunAll gives the
> following:
> Overall time 57h 52m 41.188s (
> - Build Net Time: 45h 10m 31.686s,
> - Tests: 38h 58m 30.8s,
> - Src. Update: 2h 29m 51.14s,
> - Artifacts Publishing: 26m 38.295s,
> - Dependencies Resolving: 9h 45m 1.809s,
> - Timeouts: 3h 54m 16.071s)
>
> Currently, Efficiency is around 67-72%
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 29 апр. 2019 г. в 12:57, Nikolay Izhikov :
>
> > Ivan,
> >
> > I thought by "faster" we all meant time between One push RunAll button and
> > one get results.
> > What else definition are possible here?
> >
> > В Пн, 29/04/2019 в 12:04 +0300, Павлухин Иван пишет:
> > > Nikolay,
> > >
> > > > Why we should imagine it?
> > >
> > > Only in order to understand what do we mean by "faster". My question was:
> > >
> > > 1st approach will take less agent time in sum. 2nd approach will
> > > complete faster in wall clock time. And your main concern is "total
> > > agent time". Am I right?
> > >
> > > пн, 29 апр. 2019 г. в 12:01, Nikolay Izhikov :
> > > >
> > > > > Let's imagine that we have an infinite number of agents
> > > >
> > > > Why we should imagine it?
> > > > We don't have infinite number of agents.
> > > >
> > > > And we have several concurrent Run All.
> > > >
> > > >
> > > > В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет:
> > > > > Vyacheslav,
> > > > >
> > > > > I finally figured out that "faster" means "total agent time".
> > > > >
> > > > > Let's imagine that we have an infinite number of agents. And 2
> > approaches:
> > > > > 1. Uber "Build Apache Ignite" containing all checks.
> > > > > 2. Separate jobs for compilation, checkstyle and etc.
> > > > >
> > > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > > complete faster in wall clock time. And your main concern is "total
> > > > > agent time". Am I right?
> > > > >
> > > > > пн, 29 апр. 2019 г. в 11:42, Vyacheslav Daradur  > >:
> > > > > >
> > > > > > Hi, Ivan,
> > > > > >
> > > > > > We are in the thread "Make TC Run All faster", so the main benefit
> > is
> > > > > > to make TC faster :)
> > > > > >
> > > > > > Advantages:
> > > > > > - 1 TC agent required instead of 4;
> > > > > > - RunAll will be faster, in case of builds queue;
> > > > > >
> > > > > > Also, the "licenses" profile is included in the step of a release
> > > > > > build. I believe check-style also should be included.
> > > > > >
> > > > > > The generation of Javadocs is an optional step at preparing the
> > > > > > release, but its check on TC takes significant time in case of the
> > > > > > separate build.
> > > > > >
> > > > > > > > Returning to "Build Apache Ignite" it seems to me that
> > ideally, it can
> > > > > >
> > > > > > be hierarchical.
> > > > > >
> > > > > > I agree, all the checks may be set as a separate step in the
> > build's
> > > > > > configuration. This helps with the main problem I'm trying to
> > solve -
> > > > > > resolving of dependencies which takes the most time of the builds.
> > > > > >
> > > > > > On Mon, Apr 29, 2019 at 11:24 AM Павлухин Иван <
> > vololo...@gmail.com> wrote:
> > > > > > >
> > > > > > > Vyacheslav, Maxim,
> > > > > > >
> > > > > > > Can we once again outline what benefits aggregated "Build Apache
> > > > > > > Ignite" performing various checks has comparing to a modularized
> > > > > > > approach in which separate builds perform separate tasks?
> > > > > > >
> > > > > > > For example, modularized approach looks nice because it is
> > similar to
> > > > > > > good practices in software development where we separate
> > > > > > 

Re: Brainstorm: Make TC Run All faster

[Павлухин Иван]

Hi Dmitriy,

Thank you for adding staistics! With a good measure we can easily
compare different setups.

ср, 1 мая 2019 г. в 08:34, Павлухин Иван :
>
> Nikolay,
>
> > I thought by "faster" we all meant time between One push RunAll button and 
> > one get results.
> > What else definition are possible here?
>
> But this "faster" depends on a current load. E.g. if we have a lot of
> idle slaves then incresead parallelism will allow a build to complete
> "faster". If we have many slaves busy than parallelism might "slow"
> down an execution. I predict that you are mostly interested in average
> metrics when TC has a build queue as it usually have on weekdays. If
> so then aggregatin other related jobs into "Build Apache Ignite" might
> speed execution of RunAll measured in wall clock time.
>
> And also I would like to return to a point I already mentioned earlier
> in other conversations. Our CI environment suites perfectly for an
> agile way of working. I suppose that we could try different setups
> almost freely and see if it gives a desired effect (and rollback if
> no). I am ok to go with Vyacheslav proposal. But I would like to
> observe some metrics showing that builds become running faster than
> before. Can we do this?
>
> вт, 30 апр. 2019 г. в 17:42, Dmitriy Pavlov :
> >
> > Hi, One more metric we can introduce is efficiency. Test run time/overall
> > time. The goal of launching RunAll is to obtain tests results.
> >
> > I've added more statistics to be displayed in the TC Bot, so now you may
> > click 'more' button near 'chain results' caption and you'll see sum of
> > statistics for all chain or individual suite. Latest RunAll gives the
> > following:
> > Overall time 57h 52m 41.188s (
> > - Build Net Time: 45h 10m 31.686s,
> > - Tests: 38h 58m 30.8s,
> > - Src. Update: 2h 29m 51.14s,
> > - Artifacts Publishing: 26m 38.295s,
> > - Dependencies Resolving: 9h 45m 1.809s,
> > - Timeouts: 3h 54m 16.071s)
> >
> > Currently, Efficiency is around 67-72%
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 29 апр. 2019 г. в 12:57, Nikolay Izhikov :
> >
> > > Ivan,
> > >
> > > I thought by "faster" we all meant time between One push RunAll button and
> > > one get results.
> > > What else definition are possible here?
> > >
> > > В Пн, 29/04/2019 в 12:04 +0300, Павлухин Иван пишет:
> > > > Nikolay,
> > > >
> > > > > Why we should imagine it?
> > > >
> > > > Only in order to understand what do we mean by "faster". My question 
> > > > was:
> > > >
> > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > complete faster in wall clock time. And your main concern is "total
> > > > agent time". Am I right?
> > > >
> > > > пн, 29 апр. 2019 г. в 12:01, Nikolay Izhikov :
> > > > >
> > > > > > Let's imagine that we have an infinite number of agents
> > > > >
> > > > > Why we should imagine it?
> > > > > We don't have infinite number of agents.
> > > > >
> > > > > And we have several concurrent Run All.
> > > > >
> > > > >
> > > > > В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет:
> > > > > > Vyacheslav,
> > > > > >
> > > > > > I finally figured out that "faster" means "total agent time".
> > > > > >
> > > > > > Let's imagine that we have an infinite number of agents. And 2
> > > approaches:
> > > > > > 1. Uber "Build Apache Ignite" containing all checks.
> > > > > > 2. Separate jobs for compilation, checkstyle and etc.
> > > > > >
> > > > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > > > complete faster in wall clock time. And your main concern is "total
> > > > > > agent time". Am I right?
> > > > > >
> > > > > > пн, 29 апр. 2019 г. в 11:42, Vyacheslav Daradur  > > >:
> > > > > > >
> > > > > > > Hi, Ivan,
> > > > > > >
> > > > > > > We are in the thread "Make TC Run All faster", so the main benefit
> > > is
> > > > > > > to make TC faster :)
> > > > > > >
> > > > > > > Advantages:
> > > > > > > - 1 TC agent required instead of 4;
> > > > > > > - RunAll will be faster, in case of builds queue;
> > > > > > >
> > > > > > > Also, the "licenses" profile is included in the step of a release
> > > > > > > build. I believe check-style also should be included.
> > > > > > >
> > > > > > > The generation of Javadocs is an optional step at preparing the
> > > > > > > release, but its check on TC takes significant time in case of the
> > > > > > > separate build.
> > > > > > >
> > > > > > > > > Returning to "Build Apache Ignite" it seems to me that
> > > ideally, it can
> > > > > > >
> > > > > > > be hierarchical.
> > > > > > >
> > > > > > > I agree, all the checks may be set as a separate step in the
> > > build's
> > > > > > > configuration. This helps with the main problem I'm trying to
> > > solve -
> > > > > > > resolving of dependencies which takes the most time of the builds.
> > > > > > >
> > > > > > > On Mon, Apr 29, 2019 at 11:24 AM Павлухин Иван <
> > > vololo...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Vyacheslav, Maxim,
> > > > > > > >
> > > > > > > 

Re: AI 3.0: writeSynchronizationMode re-thinking

[Sergey Kozlov]

Anton

I'm Ok with your proposal but IMO  it should be provided as IEP?

On Mon, Apr 29, 2019 at 4:05 PM Anton Vinogradov  wrote:

> Sergey,
>
> I'd like to continue the discussion since it closely linked to the problem
> I'm currently working on.
>
> 1) writeSynchronizationMode should not be a part of cache configuration,
> agree.
> This should be up to the user to decide how strong should be "update
> guarantee".
> So, I propose to have a special cache proxy, .withBlaBla() (at 3.x).
>
> 2) Primary fail on !FULL_SYNC is not the single problem leads to an
> inconsistent state.
> Bugs and incorrect recovery also cause the same problem.
>
> Currently, we have a solution [1] to check cluster to be consistent, but it
> has a bad resolution (will tell you only what partitions are broken).
> So, to find the broken entries you need some special API, which will check
> all copies and let you know what's went wrong.
>
> 3) Since we mostly agree that write should affect some backups in sync way,
> how about to have similar logic for reading?
>
> So, I propose to have special proxy .withQuorumRead(backupsCnt) which will
> check the explicit amount of backups on each read and return you the latest
> values.
> This proxy already implemented [2] for all copies, but I'm going to extend
> it with explicit backups number.
>
> Thoughts?
>
> 3.1) Backups can be checked in two ways:
> - request data from all backups, but wait for explicit number (solves the
> slow backup issue, but produce traffic)
> - request data from an explicit number of backups (less traffic, but can be
> as slow as all copies check case)
> what strategy is better? Should it be configurable?
>
> [1]
>
> https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums
> [2] https://issues.apache.org/jira/browse/IGNITE-10663
>
> On Thu, Apr 25, 2019 at 7:04 PM Sergey Kozlov 
> wrote:
>
> > There's another point to improve:
> > if  *syncPartitions=N* comes as the configurable in run-time it will
> allow
> > to manage the consistency-performance balance runtime, e.g. switch to
> full
> > async for preloading and then go to back to full sync for regular
> > operations
> >
> >
> > On Thu, Apr 25, 2019 at 6:48 PM Sergey Kozlov 
> > wrote:
> >
> > > Vyacheskav,
> > >
> > > You're right with the referring to MongoDB doc. In general the idea is
> > > very similar. Many vendors use such approach (1).
> > >
> > > [1]
> > >
> >
> https://dev.mysql.com/doc/refman/8.0/en/replication-options-master.html#sysvar_rpl_semi_sync_master_wait_for_slave_count
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Apr 25, 2019 at 6:40 PM Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > >
> > >> Hi, Sergey,
> > >>
> > >> Makes sense to me in case of performance issues, but may lead to
> losing
> > >> data.
> > >>
> > >> >> *by the new option *syncPartitions=N* (not best name just for
> > >> referring)
> > >>
> > >> Seems similar to "Write Concern"[1] in MongoDB. It is used in the same
> > >> way as you described.
> > >>
> > >> On the other hand, if you have such issues it should be investigated
> > >> first: why it causes performance drops: network issues etc.
> > >>
> > >> [1] https://docs.mongodb.com/manual/reference/write-concern/
> > >>
> > >> On Thu, Apr 25, 2019 at 6:24 PM Sergey Kozlov 
> > >> wrote:
> > >> >
> > >> > Ilya
> > >> >
> > >> > See comments inline.
> > >> > On Thu, Apr 25, 2019 at 5:11 PM Ilya Kasnacheev <
> > >> ilya.kasnach...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hello!
> > >> > >
> > >> > > When you have 2 backups and N = 1, how will conflicts be resolved?
> > >> > >
> > >> >
> > >> > > Imagine that you had N = 1, and primary node failed immediately
> > after
> > >> > > operation. Now you have one backup that was updated synchronously
> > and
> > >> one
> > >> > > which did not. Will they stay unsynced, or is there any mechanism
> of
> > >> > > re-syncing?
> > >> > >
> > >> >
> > >> > Same way as Ignite processes the failures for PRIMARY_SYNC.
> > >> >
> > >> >
> > >> > >
> > >> > > Why would one want to "update for 1 primary and 1 backup
> > >> synchronously,
> > >> > > update the rest of backup partitions asynchronously"? What's the
> use
> > >> case?
> > >> > >
> > >> >
> > >> > The case to have more backups but do not pay the performance penalty
> > for
> > >> > that :)
> > >> > For the distributed systems one backup looks like risky. But more
> > >> backups
> > >> > directly impacts to performance.
> > >> > Other point is to split the strict consistent apps like bank apps
> and
> > >> the
> > >> > other apps like fraud detection, analytics, reports and so on.
> > >> > In that case you can configure partitions distribution by a custom
> > >> affinity
> > >> > and have following:
> > >> >  - first set of nodes for critical (from consistency point)
> operations
> > >> >  - second set of nodes have async backup partitions only for other
> > >> > operations (reports, analytics)
> > >> >
> > 

[jira] [Created] (IGNITE-11827) Scalar doesn't support anonymous message listeners

[Alex Savitsky (JIRA)]

Alex Savitsky created IGNITE-11827:
--

 Summary: Scalar doesn't support anonymous message listeners
 Key: IGNITE-11827
 URL: https://issues.apache.org/jira/browse/IGNITE-11827
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.7
Reporter: Alex Savitsky


This works:


{code:scala}
import java.util.UUID

import org.apache.ignite.configuration.IgniteConfiguration
import org.apache.ignite.lang.IgniteBiPredicate
import org.apache.ignite.scalar.lang.ScalarPredicate2Function
import org.apache.ignite.scalar.scalar
import org.apache.ignite.scalar.scalar._

object TestIgniteMessagingScala extends App {
  class ScalaListener extends ScalarPredicate2Function[UUID, String](new 
IgniteBiPredicate[UUID, String] {
override def apply(nodeId: UUID, msg: String): Boolean = {
  System.out.println("Received ordered message [msg=" + msg + ", from=" + 
nodeId + ']')
  true
}
  })
  scalar(new 
IgniteConfiguration().setClientMode(true).setPeerClassLoadingEnabled(true)) {
val messaging = ignite$.message(ignite$.cluster.forRemotes)
messaging.remoteListen("MyUnOrderedTopic", new ScalaListener)
for (i <- 1 to 10)
  messaging.send("MyUnOrderedTopic", Integer.toString(i))
  }
}
{code}

However, trying to define the same listener in place, fails:

{code:scala}
import java.util.UUID

import org.apache.ignite.configuration.IgniteConfiguration
import org.apache.ignite.lang.IgniteBiPredicate
import org.apache.ignite.scalar.lang.ScalarPredicate2Function
import org.apache.ignite.scalar.scalar
import org.apache.ignite.scalar.scalar._

object TestIgniteMessagingScala extends App {
  scalar(new 
IgniteConfiguration().setClientMode(true).setPeerClassLoadingEnabled(true)) {
val messaging = ignite$.message(ignite$.cluster.forRemotes)
val listener: IgniteBiPredicate[UUID, String] = (nodeId: UUID, msg: String) 
=> {
  System.out.println("Received ordered message [msg=" + msg + ", from=" + 
nodeId + ']')
  true
}
messaging.remoteListen("MyUnOrderedTopic", listener)
for (i <- 1 to 10)
  messaging.send("MyUnOrderedTopic", Integer.toString(i))
  }
}
{code}

The exception is:


{noformat}
Exception in thread "main" class 
org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object 
[typeName=org.apache.ignite.scalar.lang.ScalarPredicate2]
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
at 
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
at 
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
at 
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
at 
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
at 
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10169)
at 
org.apache.ignite.internal.GridMessageListenHandler.p2pUnmarshal(GridMessageListenHandler.java:194)
at 
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1362)
at 
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:111)
at 
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:203)
at 
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:194)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:727)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read 
field [name=p]
at 
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
... 18 more
Caused by: class 

Re: Brainstorm: Make TC Run All faster

[Dmitriy Pavlov]

Hi, One more metric we can introduce is efficiency. Test run time/overall
time. The goal of launching RunAll is to obtain tests results.

I've added more statistics to be displayed in the TC Bot, so now you may
click 'more' button near 'chain results' caption and you'll see sum of
statistics for all chain or individual suite. Latest RunAll gives the
following:
Overall time 57h 52m 41.188s (
- Build Net Time: 45h 10m 31.686s,
- Tests: 38h 58m 30.8s,
- Src. Update: 2h 29m 51.14s,
- Artifacts Publishing: 26m 38.295s,
- Dependencies Resolving: 9h 45m 1.809s,
- Timeouts: 3h 54m 16.071s)

Currently, Efficiency is around 67-72%

Sincerely,
Dmitriy Pavlov

пн, 29 апр. 2019 г. в 12:57, Nikolay Izhikov :

> Ivan,
>
> I thought by "faster" we all meant time between One push RunAll button and
> one get results.
> What else definition are possible here?
>
> В Пн, 29/04/2019 в 12:04 +0300, Павлухин Иван пишет:
> > Nikolay,
> >
> > > Why we should imagine it?
> >
> > Only in order to understand what do we mean by "faster". My question was:
> >
> > 1st approach will take less agent time in sum. 2nd approach will
> > complete faster in wall clock time. And your main concern is "total
> > agent time". Am I right?
> >
> > пн, 29 апр. 2019 г. в 12:01, Nikolay Izhikov :
> > >
> > > > Let's imagine that we have an infinite number of agents
> > >
> > > Why we should imagine it?
> > > We don't have infinite number of agents.
> > >
> > > And we have several concurrent Run All.
> > >
> > >
> > > В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет:
> > > > Vyacheslav,
> > > >
> > > > I finally figured out that "faster" means "total agent time".
> > > >
> > > > Let's imagine that we have an infinite number of agents. And 2
> approaches:
> > > > 1. Uber "Build Apache Ignite" containing all checks.
> > > > 2. Separate jobs for compilation, checkstyle and etc.
> > > >
> > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > complete faster in wall clock time. And your main concern is "total
> > > > agent time". Am I right?
> > > >
> > > > пн, 29 апр. 2019 г. в 11:42, Vyacheslav Daradur  >:
> > > > >
> > > > > Hi, Ivan,
> > > > >
> > > > > We are in the thread "Make TC Run All faster", so the main benefit
> is
> > > > > to make TC faster :)
> > > > >
> > > > > Advantages:
> > > > > - 1 TC agent required instead of 4;
> > > > > - RunAll will be faster, in case of builds queue;
> > > > >
> > > > > Also, the "licenses" profile is included in the step of a release
> > > > > build. I believe check-style also should be included.
> > > > >
> > > > > The generation of Javadocs is an optional step at preparing the
> > > > > release, but its check on TC takes significant time in case of the
> > > > > separate build.
> > > > >
> > > > > > > Returning to "Build Apache Ignite" it seems to me that
> ideally, it can
> > > > >
> > > > > be hierarchical.
> > > > >
> > > > > I agree, all the checks may be set as a separate step in the
> build's
> > > > > configuration. This helps with the main problem I'm trying to
> solve -
> > > > > resolving of dependencies which takes the most time of the builds.
> > > > >
> > > > > On Mon, Apr 29, 2019 at 11:24 AM Павлухин Иван <
> vololo...@gmail.com> wrote:
> > > > > >
> > > > > > Vyacheslav, Maxim,
> > > > > >
> > > > > > Can we once again outline what benefits aggregated "Build Apache
> > > > > > Ignite" performing various checks has comparing to a modularized
> > > > > > approach in which separate builds perform separate tasks?
> > > > > >
> > > > > > For example, modularized approach looks nice because it is
> similar to
> > > > > > good practices in software development where we separate
> > > > > > responsibilities between different classes instead of
> aggregating them
> > > > > > into a single class. And as usual multiple classes works together
> > > > > > coordinating by a class from upper level. So, in fact it is a
> > > > > > hierarchical structure.
> > > > > >
> > > > > > Returning to "Build Apache Ignite" it seems to me that ideally
> it can
> > > > > > be hierarchical. There is a top level compilation (assembly?)
> job but
> > > > > > it is always clear what tasks does it perform (check style, check
> > > > > > license and other subjobs).
> > > > > >
> > > > > > пт, 26 апр. 2019 г. в 17:06, Maxim Muzafarov  >:
> > > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > +1 for merging all these suites into the single one. All these
> suites
> > > > > > > (Build Apache Ignite, Javadoc, Licenses Header, Checkstyle)
> required
> > > > > > > to be `green` all the time. So, we can consider making them a
> part of
> > > > > > > build Apache Ignite procedure.
> > > > > > >
> > > > > > > Also, I'd suggest going deeper. We can try to merge `Licenses
> Header`
> > > > > > > into the `Code style checker` [1]. This will simplify the code
> > > > > > > checking process.
> > > > > > >
> > > > > > > [1] http://checkstyle.sourceforge.net/config_header.html
> > > > > > >
> > > > > > > On 

[jira] [Created] (IGNITE-11826) Unwrap Option[T] types for @QuerySqlField for known SQL types

[Alex Savitsky (JIRA)]

Alex Savitsky created IGNITE-11826:
--

 Summary: Unwrap Option[T] types for @QuerySqlField for known SQL 
types
 Key: IGNITE-11826
 URL: https://issues.apache.org/jira/browse/IGNITE-11826
 Project: Ignite
  Issue Type: Improvement
  Components: binary, sql
Affects Versions: 2.7
Reporter: Alex Savitsky


Currently, annotating an Option type with {{@QuerySqlField}} results in a DB 
column of type {{OTHER}}. It would be nice if known SQL types could be 
unwrapped from their optionals, so that, for example, {{@QuerySqlField name: 
Option[String]}} would result in a field of type {{VARCHAR}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11825) Test GridCommandHandlerTest#testCacheIdleVerifyNodeFilter fails with "Duplicate row in index"

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-11825:


 Summary: Test GridCommandHandlerTest#testCacheIdleVerifyNodeFilter 
fails with "Duplicate row in index"
 Key: IGNITE-11825
 URL: https://issues.apache.org/jira/browse/IGNITE-11825
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev


A freshly contributed test fails around half of runs with exceptions like:

{code}
[2019-04-30 
14:15:14,355][ERROR][data-streamer-stripe-0-#20402%gridCommandHandlerTest0%][IgniteTestResources]
 Failed to set initial value for cache entry: DataStreamerEntry [key=UserKeyCach
eObjectImpl [part=25, val=25, hasValBytes=true], val=UserCacheObjectImpl 
[val=25, hasValBytes=true]]
class 
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
 Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=25, 
val=25, hasValBytes=tru
e], hash=25, cacheId=0]
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1817)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1619)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1602)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2160)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:433)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4282)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3430)
at 
org.apache.ignite.internal.processors.cache.GridCacheEntryEx.initialValue(GridCacheEntryEx.java:772)
at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2280)
at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6845)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:550)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Duplicate row in index.
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:437)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:423)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5643)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5629)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:359)
at 
org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:285)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11400(BPlusTree.java:92)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:3622)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7100(BPlusTree.java:3302)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.onNotFound(BPlusTree.java:3860)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$5800(BPlusTree.java:3652)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1902)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1784)
{code}

which will lead to partition divergence.

Maybe node filter is not implemented correctly, but otherwise it would look 
like a Durable Memory bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11824) Integrate PageLockTracker to DataStructure (per-thread tracker)

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-11824:
---

 Summary: Integrate PageLockTracker to DataStructure (per-thread 
tracker)
 Key: IGNITE-11824
 URL: https://issues.apache.org/jira/browse/IGNITE-11824
 Project: Ignite
  Issue Type: Sub-task
Reporter: Dmitriy Govorukhin
Assignee: Dmitriy Govorukhin


After [IGNITE-11750] will be completed, we will have a structure for tracking 
page locks per-thread. The next step, need to integrate it into diagnostic API 
and implements a component for creating this structure per-thread.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: "Idle verify" to "Online verify"

[Anton Vinogradov]

Ivan,

Thanks for the detailed explanation.
I'll try to implement the PoC to check the idea.

On Mon, Apr 29, 2019 at 8:22 PM Ivan Rakov  wrote:

> > But how to keep this hash?
> I think, we can just adopt way of storing partition update counters.
> Update counters are:
> 1) Kept and updated in heap, see
> IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#pCntr (accessed during
> regular cache operations, no page replacement latency issues)
> 2) Synchronized with page memory (and with disk) on every checkpoint,
> see GridCacheOffheapManager#saveStoreMetadata
> 3) Stored in partition meta page, see PagePartitionMetaIO#setUpdateCounter
> 4) On node restart, we init onheap counter with value from disk (for the
> moment of last checkpoint) and update it to latest value during WAL
> logical records replay
>
> > 2) PME is a rare operation on production cluster, but, seems, we have
> > to check consistency in a regular way.
> > Since we have to finish all operations before the check, should we
> > have fake PME for maintenance check in this case?
>  From my experience, PME happens on prod clusters from time to time
> (several times per week), which can be enough. In case it's needed to
> check consistency more often than regular PMEs occur, we can implement
> command that will trigger fake PME for consistency checking.
>
> Best Regards,
> Ivan Rakov
>
> On 29.04.2019 18:53, Anton Vinogradov wrote:
> > Ivan, thanks for the analysis!
> >
> > >> With having pre-calculated partition hash value, we can
> > automatically detect inconsistent partitions on every PME.
> > Great idea, seems this covers all broken synс cases.
> >
> > It will check alive nodes in case the primary failed immediately
> > and will check rejoining node once it finished a rebalance (PME on
> > becoming an owner).
> > Recovered cluster will be checked on activation PME (or even before
> > that?).
> > Also, warmed cluster will be still warmed after check.
> >
> > Have I missed some cases leads to broken sync except bugs?
> >
> > 1) But how to keep this hash?
> > - It should be automatically persisted on each checkpoint (it should
> > not require recalculation on restore, snapshots should be covered too)
> > (and covered by WAL?).
> > - It should be always available at RAM for every partition (even for
> > cold partitions never updated/readed on this node) to be immediately
> > used once all operations done on PME.
> >
> > Can we have special pages to keep such hashes and never allow their
> > eviction?
> >
> > 2) PME is a rare operation on production cluster, but, seems, we have
> > to check consistency in a regular way.
> > Since we have to finish all operations before the check, should we
> > have fake PME for maintenance check in this case?
> >
> > On Mon, Apr 29, 2019 at 4:59 PM Ivan Rakov  > > wrote:
> >
> > Hi Anton,
> >
> > Thanks for sharing your ideas.
> > I think your approach should work in general. I'll just share my
> > concerns about possible issues that may come up.
> >
> > 1) Equality of update counters doesn't imply equality of
> > partitions content under load.
> > For every update, primary node generates update counter and then
> > update is delivered to backup node and gets applied with the
> > corresponding update counter. For example, there are two
> > transactions (A and B) that update partition X by the following
> > scenario:
> > - A updates key1 in partition X on primary node and increments
> > counter to 10
> > - B updates key2 in partition X on primary node and increments
> > counter to 11
> > - While A is still updating another keys, B is finally committed
> > - Update of key2 arrives to backup node and sets update counter to 11
> > Observer will see equal update counters (11), but update of key 1
> > is still missing in the backup partition.
> > This is a fundamental problem which is being solved here:
> > https://issues.apache.org/jira/browse/IGNITE-10078
> > "Online verify" should operate with new complex update counters
> > which take such "update holes" into account. Otherwise, online
> > verify may provide false-positive inconsistency reports.
> >
> > 2) Acquisition and comparison of update counters is fast, but
> > partition hash calculation is long. We should check that update
> > counter remains unchanged after every K keys handled.
> >
> > 3)
> >
> >> Another hope is that we'll be able to pause/continue scan, for
> >> example, we'll check 1/3 partitions today, 1/3 tomorrow, and in
> >> three days we'll check the whole cluster.
> > Totally makes sense.
> > We may find ourselves into a situation where some "hot" partitions
> > are still unprocessed, and every next attempt to calculate
> > partition hash fails due to another concurrent update. We should
> > be able to track progress of validation (% of calculation time
> > wasted due to 

Re: [IEP-35] Monitoring & Profiling. Proof of concept

[Maxim Muzafarov]

Hello Nikolay,

I've looked through your PRs changes.

> Sensors

How will be recorded throughput sensor values which will require an
interval for the rate calculations? Do we have such an example? For
instance, getAllocationRate() or getEvictionRate(). These metrics are
out of the scope of current PoC and IEP as they are not related to the
user metrics, but it is a good example of a particular metric type.

It seems to me that we can add an additional parameter of
`sensitivityLevel` to provide for the user a flexible sensor control
(e.g., INFO, WARN, NOTICE, DEBUG).

It also seems that for the sensors getValue() the completely
functional java approach can be used. Am I right?

On Mon, 29 Apr 2019 at 11:44, Nikolay Izhikov  wrote:
>
> Hello, Vyacheslav.
>
> Thanks for the feedback!
>
> > HttpExposer with Jetty's dependencies should be detached> from the core 
> > module.
>
> Agreed. module hierarchy is the essence of the next steps.
> For now it just a proof of my ideas for Ignite monitoring we can discuss.
>
> > I like your approach with 'wrapper' for monitored objects, like don't like 
> > using 'ServiceConfiguration' directly as a monitored object for services
>
> Agreed in general.
> Seems, choosing the right data to expose is the matter of separate discussion 
> for each Ignite entities.
> I've planned to file tickets for each entity so anyone interested can share 
> his vision in it.
>
> > In my opinion, each sensor should have a timestamp.
>
> I'm not sure that *every* sensor should have directly associated timestamp.
> Seems, we should support sensors without timestamp for a current monitoring 
> numbers at least.
>
> > Also, it'd be great to have an ability to store a list of a fixed size> of 
> > last N sensors
>
> What use-cases do you know for such sensors?
> We have plans to support fixed size lists to show "Last N SQL queries" or 
> similar data.
> Essentially, a sensor is just a single value with the name and known meaning.
>
> > It'd be great if you provide a more extended test to show the work of> the 
> > system.
>
> Sorry, for that :)
> When you run 'MonitoringSelfTest' you should open 
> http://localhost:8080/ignite/monitoring to view exposed info.
> I provide this info in gist - 
> https://gist.github.com/nizhikov/aa1e6222e6a3456472b881b8deb0e24d
>
> I will extend this test to print results to console in the next iterations - 
> stay tuned :)
>
> В Вс, 28/04/2019 в 23:35 +0300, Vyacheslav Daradur пишет:
> > Hi, Nikolay,
> >
> > I looked through PR and IEP, and I have some comments:
> >
> > It would be better to implement it as a separate module, I can't say
> > if it is possible for the main part of monitoring or not, but I
> > believe that HttpExposer with Jetty's dependencies should be detached
> > from the core module.
> >
> > I like your approach with 'wrapper' for monitored objects, like
> > 'ComputeTaskInfo' in PR, and don't like using 'ServiceConfiguration'
> > directly as a monitored object for services. I believe we shouldn't
> > mix approaches. It'd be better always use some kind of container with
> > monitored object's information to work with such data.
> >
> > In my opinion, each sensor should have a timestamp. Usually monitoring
> > systems aggregate data and build graphics according to sensors
> > timestamp.
> >
> > Also, it'd be great to have an ability to store a list of a fixed size
> > of last N sensors, not to miss them without pushing to an external
> > monitoring system.
> >
> > It'd be great if you provide a more extended test to show the work of
> > the system. Everybody who looks to PR needs to run the test and get
> > the info manually to see the completeness of sensors, this might be
> > simplified by proper test.
> >
> > Thank you!
> >
> >
> >
> > On Fri, Apr 26, 2019 at 5:56 PM Nikolay Izhikov  wrote:
> > >
> > > Hello, Igniters.
> > >
> > > I've prepared Proof of Concept for IEP-35 [1]
> > > PR can be found here - https://github.com/apache/ignite/pull/6510
> > >
> > > I've done following changes:
> > >
> > > 1. `GridMonitoringManager`  [2] - simple implementation of 
> > > manager to store all monitoring info
> > > 2. `HttpPullExposerSpi` [3] - pull exposer implementation that 
> > > can respond with JSON from http://localhost:8080/ignite/monitoring. JSON 
> > > content can be veiwed in gist [4]
> > > 3. Compute task start and finish monitoring in "compute" list [5]
> > > 4. Service registration are monitored in "service" list - [6]
> > > 5. Current `IgniteSpiMBeanAdapter` rewritten using 
> > > `GridMonitoringManager` [7]
> > >
> > > Design principles, monitoring subsystem details and new Ignite entities 
> > > can be found in IEP [1].
> > >
> > > My next steps will be:
> > >
> > > 1. Implementation of JMX exposer
> > > 2. Registration of all "lists" and "sensor groups" as a SQL 
> > > System view.
> > > 3. Add monitoring for all unmonitoring Ignite API. (described in 
> > > IEP).
> > >  

Re: [ML] Machine Learning Pipeline Improvement

[Manu]

Hi, all!

Could be viable to integrate Apache Arrow to improve ML computation using
GPU?
Out of this thread, could be viable to integrate Apache Arrow to improve
Indexing computation using GPU?

Regards

https://rapids.ai   
https://arrow.apache.org   



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Thin client: transactions support

[Alex Plehanov]

Hello, Igniters!

I've update IEP [1] and implement PoC according to new approach (multiple
concurrent transactions per connection).
But to move forward another feature need to be implemented: suspend/resume
for pessimistic transactions (IGNITE-5714 [2]). Implementation of
suspend/resume is ready now and ticket in 'Patch available' status. Can any
transactions expert help with review of IGNITE-5714?

[1]:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support
[2]: https://issues.apache.org/jira/browse/IGNITE-5714

чт, 4 апр. 2019 г. в 11:32, Alex Plehanov :

> Vladimir,
>
> Ok, then I will rewrite IEP in the near future.
>
> чт, 4 апр. 2019 г. в 11:14, Vladimir Ozerov :
>
>> Hi Alex,
>>
>> I think we should be able to handle many transactions through a single
>> connection. This will make our protocol and client implementations much
>> more efficient, and simplicity from developer's perspective is not our
>> goal. Consider normal nodes. We have server nodes and client nodes. You
>> may
>> span whatever number of transactions you need, but all of them are
>> coordinated through a single connection. The same should be applicable to
>> thin clients. Protocol is already designed to handle this, as we pass
>> unique operation ID in order to distinguish one operation from another. It
>> is true, though, that we will have to introduce a kind of "session"
>> concept, and pass additional identifier along with cache operations, but
>> this doesn't sound like a problem to me.
>>
>> And provided that currently server-side transactions are bound to threads
>> artificially, I would say that the first step in implementation of
>> transactions on thin clients should be decoupling server-side transactions
>> from threads. Without this we will have very inefficient implementation,
>> when every new client transaction have to spawn a new thread. This is slow
>> and introduces high memory pressure on a cluster node. We already work
>> this
>> way for MVCC transactions which are spawned from JDBC driver, and believe
>> me, we do not want to replicated this bad practice to other clients :-)
>>
>> Vladimir.
>>
>> On Thu, Apr 4, 2019 at 10:08 AM Alex Plehanov 
>> wrote:
>>
>> > Guys, so, do we need multiple concurrent transactions per connection?
>> >
>> > There are pros and cons for each approach. Difference between
>> approaches:
>> >
>> > One transaction at a time per connection:
>> >  - This approach is used in RDBMS world and users got used to it
>> >  - To use transactions concurrently users need to use different
>> connections
>> > and get these connections via something like a connection pool
>> >  - Easy to implement (in fact, PoC is already done)
>> >
>> > Multiple concurrent transactions per connection:
>> >  - At least for java thin client, we can implement transaction per
>> thread
>> > approach as implemented now for the thick client (perhaps other thin
>> > clients can implement the same abstraction)
>> >  - There is also protocol change for all cache operations needed (to
>> bind
>> > cache operation to the transaction)
>> >  - Significant changes to all implemented clients are needed
>> >  - Implementation on the server side is more complex
>> >
>> > What do you think?
>> >
>> >
>> > вт, 2 апр. 2019 г. в 16:29, Alex Plehanov :
>> >
>> > > Ilya,
>> > >
>> > > > We should be able to multiplex several transactions using a single
>> > > Client connection.
>> > > In this case, we should significantly change cache operations syntax
>> (for
>> > > each implemented client), to bind each operation to the transaction.
>> > >
>> > > > I want to also ask if "Number of entries participating in
>> transaction
>> > > (may be approximate). 0 - default value." is needed.
>> > > I've tried to minimize API changes between thick and thin client to
>> > > simplify move from one to another. It's the only reason. But I agree
>> with
>> > > you, the parameter is not very useful.
>> > >
>> > >
>> > > вт, 2 апр. 2019 г. в 14:48, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com>:
>> > >
>> > >> Hello!
>> > >>
>> > >> Pavel, I agree with you thorougly. We should be able to multiplex
>> > several
>> > >> transactions using a single Client connection. This means adding
>> > >> Transaction id parameter to every affected cache operation / SQL
>> > statement
>> > >> (if applicable) to make sure we do cache operations on relevant
>> > >> transaction.
>> > >>
>> > >> This is how other things work in Ignite, such as communication. We do
>> > not
>> > >> open dozens of connections, we multiplex operations asynchronously
>> > through
>> > >> a single connection.
>> > >>
>> > >> I think that trying to pool Ignite connections will be highly
>> > >> inconvenient,
>> > >> since there is no existing infrastructure for such pooling (like
>> there
>> > >> exists for JDBC).
>> > >>
>> > >> I want to also ask if "Number of entries participating in transaction
>> > (may
>> > >> be approximate). 0 - default value." is 

Re: IgniteConfigVariationsAbstractTest subclasses do not run

[Ivan Fedotov]

Ivan R., Ivan P., thank you for the suggestions, I took a look at them.

According to checking CacheAtomicityMode on null in
IgniteCacheConfigVariationsAbstractTest#atomicityMode - are you sure
that checking should proceed on test level? May be it will be better to
make default cache value in case if cc.getAtomicityMode() == null
in CacheConfiguration constructor [1]?

Also let me suggest one minor change according cache specification in
IgniteCacheReadThroughEvictionSelfTest. The same logic is used in
GridCacheAbstractSelfTest#cacheConfiguration [2].
I think we can excract this code block in a separate static methods
(initCacheConfig, for example) in IgniteCacheReadThroughEvictionSelfTest and
invoke it in IgniteCacheReadThroughEvictionSelfTest#variationConfig and
GridCacheAbstractSelfTest#cacheConfiguration methods.

In addition, I want to draw your attention to
IgniteContinuousQueryConfigVariationsSuite test runs [3].
CacheContinuousQueryVariationsTest are generated dynamically.
The first batch (12 tests) works fine, but on the next runs we got NPE in
IgniteCacheConfigVariationsAbstractTest#afterTest - default cache does not
exist and we can not
invoke unwrap() on this [4].
Seems that the problem is in
IgniteCacheConfigVariationsAbstractTest#beforeTestsStarted/afterTestsStopped
methods, cache is not properly created (or already existed cache is
destroyed).

By the way, should I resolve these issues in the context of IGNITE-11708 or
create a separate ticket(s)?

[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/CacheConfiguration.java#L434
[2]
https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridCacheAbstractSelfTest.java#L235
[3]
https://ci.ignite.apache.org/viewLog.html?buildId=3701865=IgniteTests24Java8_RunAllNightly
[4]
https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/testframework/junits/IgniteCacheConfigVariationsAbstractTest.java#L212

пт, 26 апр. 2019 г. в 18:01, Ivan Rakov :

> Ivan P.,
>
> Good catch, thanks.
>
> I was wrong, test scenario is correct. The problem was in
> atomicityMode() method - it could have returned null (which was okay for
> config generation, but wasn't expected in the test code).
> Please take a look at tx_out_test_fixed.patch (attached to
> IGNITE-11708). To sum it up, both issues should be fixed now.
>
> Best Regards,
> Ivan Rakov
>
> On 26.04.2019 14:40, Павлухин Иван wrote:
> > Ivan R.,
> >
> > As I can see IgniteCacheConfigVariationsFullApiTest#testGetOutTx does
> > not expect lock/unlock events due to line:
> > if (atomicityMode() == ATOMIC)
> >  return lockEvtCnt.get() == 0;
> >
> > Could you please elaborate?
> >
> > пт, 26 апр. 2019 г. в 13:32, Ivan Rakov :
> >> Ivan,
> >>
> >> Seems like IgniteCacheReadThroughEvictionSelfTest is broken. Test
> >> scenario assumes that even after expiration entry will be present in
> >> IgniteCache as per it will be loaded from CacheStore. However,
> >> CacheStore is not specified in node config. I've added patch that
> >> enables cache store factory, please check IGNITE-11708 attachments.
> >> Regarding IgniteCacheConfigVariationsFullApiTest#testGetOutTx* tests:
> >> from my point of view, test scenarios make no sense. We perform get()
> >> operation from ATOMIC caches and expect that entries will be locked. I
> >> don't understand why we should lock entries on ATOMIC get, therefore I
> >> suppose to remove part of code where we listen and check
> >> EVT_CACHE_OBJECT_LOCKED/UNLOCKED events.
> >>
> >> Best Regards,
> >> Ivan Rakov
> >>
> >> On 17.04.2019 22:05, Ivan Rakov wrote:
> >>> Hi Ivan,
> >>>
> >>> I've checked your branch. Seems like these tests fail due to real
> >>> issue in functionality.
> >>> I'll take a look.
> >>>
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>> On 17.04.2019 13:54, Ivan Fedotov wrote:
>  Hi, Igniters!
> 
>  During work on iep-30[1] I discovered that
>  IgniteConfigVariationsAbstractTest subclasses - it is about 15_000
>  tests[2]
>  - do not work.
>  You can check it just run one of the tests with log output, for
> example
>  ConfigVariationsTestSuiteBuilderTest#LegacyLifecycleTest#test1 [3].
>  There is no warning notification in the console. The same situation
> with
>  other IgniteConfigVariationsAbstractTest subclasses - tests run, but
>  they
>  simply represent empty code.
> 
>  So, I created a ticket on such issue [4] and it turned out that the
>  problem
>  is with ruleChain in IgniteConfigVariationsAbstractTest [5].
>  The rule that is responsible for running a test statement does not
> start
>  indeed [6] under ruleChain runRule. I suggested a solution - move
>  testsCfg
>  initialization to
>  IgniteConfigVariationsAbstractTest#beforeTestsStarted method. After
> such
>  changes ruleChain becomes not necessary.
> 
>  But I