Re: CountDownLatch issue in Ignite 2.6 version

[Prasad Bhalerao]

I just checked the ignite doc for atomic configuration.
But it doesn't say that it is applicable to distributed data structures.

Is it really applicable to distributed data structures like count down latch

On Tue 9 Jun, 2020, 7:26 AM Prasad Bhalerao  Hi,
> I was under the impression that countdown latch is implemented in
> replicated cache. So when any number of nodes go down it does not loose
> it's state.
>
> Can you please explain why atmoc data structures are using 1 back when its
> state is very important?
>
> Can we enforce  atomic data structures to use replicated cache?
>
> Which cache does ignite use to store atomic data structures?
>
> Thanks
> Prasad
>
> On Mon 8 Jun, 2020, 11:58 PM Evgenii Zhuravlev  wrote:
>
>> Hi,
>>
>> By default, cache, that stores all atomic structures has only 1 backup,
>> so, after losing all data for this certain latch, it recreates it. To
>> change the default atomic configuration use
>> IgniteConfiguration.setAtomicConfiguration.
>>
>> Evgenii
>>
>> сб, 6 июн. 2020 г. в 06:20, Akash Shinde :
>>
>>> *Issue:* Countdown latched gets reinitialize to original value(4) when
>>> one or more (but not all) node goes down. *(Partition loss happened)*
>>>
>>> We are using ignite's distributed countdownlatch to make sure that cache
>>> loading is completed on all server nodes. We do this to make sure that our
>>> kafka consumers starts only after cache loading is complete on all server
>>> nodes. This is the basic criteria which needs to be fulfilled before starts
>>> actual processing
>>>
>>>
>>>  We have 4 server nodes and countdownlatch is initialized to 4. We use
>>> cache.loadCache method to start the cache loading. When each server
>>> completes cache loading it reduces the count by 1 using countDown method.
>>> So when all the nodes completes cache loading, the count reaches to zero.
>>> When this count  reaches to zero we start kafka consumers on all server
>>> nodes.
>>>
>>>  But we saw weird behavior in prod env. The 3 server nodes were shut
>>> down at the same time. But 1 node is still alive. When this happened the
>>> count down was reinitialized to original value i.e. 4. But I am not able to
>>> reproduce this in dev env.
>>>
>>>  Is this a bug, when one or more (but not all) nodes goes down then
>>> count re initializes back to original value?
>>>
>>> Thanks,
>>> Akash
>>>
>>

Re: Continuous Queries with several remote filter on the same cache

[Denis Magda]

Roman,

Please check the following methods:
* CacheContiniousQueryHandler (the filter usage):
https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512ee54f38bb/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
* CacheContinuousQueryManager (the listener execution):
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryManager.java#L376

-
Denis


On Sun, Jun 7, 2020 at 12:19 AM  wrote:

> Hi Denis,
> A big thank you for the answer.
> Could you please tell me where can I find this logic in the sources. Which
> package should I look into?
>
> -Original Message-
> From: Denis Magda 
> Sent: Saturday, June 6, 2020 2:07 AM
> To: dev 
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Hi Roman,
>
> Every continuous query is a unique entity that is processed by servers
> independently. With your example, the server node will execute all 20
> filters for every cache insert/update operation. The server will notify
> through local listeners only those clients whose remote filters returned
> 'true'.
>
> -
> Denis
>
>
> On Thu, Jun 4, 2020 at 8:44 PM  wrote:
>
> > Hi Community,
> >
> > I ask this question here because I haven't found the answer in the
> > documentation.
> >
> > Could you please clarify how Continuous Queries work? What the
> > behavior of Continuous Queries if we have several clients with
> > different Remote Filters on the same cache? For example, if we have:
> > one server node with cache and we have up to 20 client nodes each of
> > them will execute Continuous Query on the same cache but with
> > different Remote Filters. Will each client get the data according to
> > its remote filter? Or it is supposed to have only one Remote Filter
> > for all clients and every client should filter data in its local event
> listener?
> > I would be grateful if you send some link which describes the behavior
> > of Continuous Queries more thoroughly.
> > Best regards,
> > Roman
> >
>

[jira] [Created] (IGNITE-13134) Fix connection recovery timout.

[Vladimir Steshin (Jira)]

Vladimir Steshin created IGNITE-13134:
-

 Summary: Fix connection recovery timout.
 Key: IGNITE-13134
 URL: https://issues.apache.org/jira/browse/IGNITE-13134
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.8.1
Reporter: Vladimir Steshin
Assignee: Vladimir Steshin


If node experiences connection issues it must establish new connection or fail 
within failureDetectionTimeout + connectionRecoveryTimout.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13133) Add integration with QuerydslPredicateExecutor for spring-data integrations

[Moldachev Sergey (Jira)]

Moldachev Sergey created IGNITE-13133:
-

 Summary: Add integration with QuerydslPredicateExecutor for 
spring-data integrations
 Key: IGNITE-13133
 URL: https://issues.apache.org/jira/browse/IGNITE-13133
 Project: Ignite
  Issue Type: New Feature
  Components: springdata
Reporter: Moldachev Sergey
Assignee: Moldachev Sergey


We have a pretty ignite-spring-data integration but we don't have a support of 
`QuerydslPredicateExecutor` which provide ability to filter cache entities by 
dynamic criterias. 

Example of usage:

{code:java}
/**
* Simple entity.
*/
@QueryEntity
public class Person {
/** First name. */
@QuerySqlField(index = true)
private String firstName;

/** Second name. */
@QuerySqlField(index = true)
private String secondName;

/** Age. **/
@QuerySqlField(index = true)
private int age;
}

/**
* Implement QuerydslPredicateExecutor interface.
*/
public interface PersonRepository extends IgniteRepository, 
QuerydslPredicateExecutor {

}

/**
* Now we can filter our entites by firstName prdicate.
*/
List persons = (List) repo.findAll(
QPerson.person.firstName.eq(firstName_1)
.or(QPerson.person.firstName.eq(firstName_2))
);
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Various shutdown guaranties

[Ivan Rakov]

Vlad, thanks for starting this discussion.

I'll try to clarify the motivation for this change as I see it.
In general, Ignite clusters are vulnerable to the data loss. Of course, we
have configurable PartitionLossPolicy, which allows to handle data loss
safely and mitigate its consequences. But being able to avoid critical
situations is always better than being able to recover from it.

The most common issue from my perspective is absence of a way to perform
rolling cluster restart safely. Scenario:
1. Backup count is 1
2. Admin wants to perform rolling restart in order to deploy new version of
business code that uses Ignite in embedded mode
3. Admin shuts down first node, replaces needed binaries and returns the
node back to the topology
4. Node joins the cluster successfully
5. Admin shuts down second node
6. Data loss happens: the second node was the only owner of a certain
partition, which was being rebalanced from the second node to the first

We can prevent such situations by introducing "safe shutdown by default"
mode, which blocks stopping node while it remains the only owner for at
least one partition. It should be applied to "common" ways of stopping
nodes - Ignite.close() and kill .
I think, option to be enabled or disabled in runtime should be a
requirement for this behavior. Safe shutdown mode has weird side-effects.
For example, admin won't be able to stop the whole cluster: stop of last
node will be blocked, because the last node is the only present owner of
all its partitions. Sure, kill -9 will resolve it, but it's still a
usability issue.

With the described dynamic property scenario will be changed as follows:
1. Admin enables "safe shutdown" mode
2. Admin shuts down first node, replaces needed binaries and returns the
node back to the topology
3. Admin shuts down second node (with either ignite.close() or kill ),
shutdown is blocked until the first node returns to the topology and
completes the rebalancing process
4. Admin proceeds the rolling restart procedure
5. Admin disables "safe shutdown" mode

This logic will also simplify the rolling restart scenario in K8S. Pod with
Ignite node won't be terminated until its termination will cause data loss.

Aside from waiting for backups, Ignition interface provide lots of options
to perform various node stop:
- Whether or not to cancel pending compute jobs
- Whether or not to perform instant halt() instead of any graceful stop
logic
- Whether or not to wait for some timeout before halt()
- Whether or not the stopped grid should be restarted
All these "stop" methods provide very custom logic. I don't see a need to
make them part of dynamic cluster-wide configuration. They still can be
invoked directly via Java API. Later we can extract some of them to dynamic
cluster-wide parameters of default stop if it will become necessary. That's
why I think we should create an enum for default shutdown policy, but only
with two options so far (we can add more later): DEFAULT and
WAIT_FOR_BACKUPS.
Regarding the "NORMAL" option that you propose (where the node is not
stopped until the rebalance is finished): I don't think that we should add
it. It doesn't ensure any strict guarantees: the data still can be lost
with it.

To sum it up, I propose:
1. Add a new method to Ignition interface to make it possible to stop with
"wait for backups" logic directly via Java API, like Ignition.stop(boolean
cancel, boolean waitForBackups)
2. Introduce "defaultShutdownPolicy" as a dynamic cluster configuration,
two values are available so far: DEFAULT and WAIT_FOR_BACKUPS
3. This property is stored in the distributed metastorage (thus persisted),
can be changed via Java API and ./control.sh
4. Behavior configured with this property will be applied only on common
ways of stopping the node - Ignite.close() and kill .
5. *Don't* add new options to the static IgniteConfiguration to avoid
conflicts between dynamic and static configuration

-- 
Best Regards,
Ivan Rakov

On Mon, Jun 8, 2020 at 6:44 PM V.Pyatkov  wrote:

> Hi
>
> We need to have ability to calling shutdown with various guaranties.
> For example:
> Need to reboot a node, but after that node should be available for
> historical rebalance (all partitions in MOVING state should have gone to
> OWNING).
>
> Implemented a circled reboot of cluster, but all data should be available
> on
> that time (at least one copy of partition should be available in cluster).
>
> Need to wait not only data available, but all jobs (before this behavior
> available through a stop(false) method invocation).
>
> All these reason required various behavior before shutting down node.
> I propose slightly modify public API and add here method which shown on
> shutdown behavior directly:
> Ignite.close(Shutdown)
>
> /public enum Shutdownn {
> /**
>  * Stop immediately as soon components are ready.
>  */
> IMMEDIATE,
> /**
>  * Stop node when all partitions completed moving from/to this node to
> another.
>  */
>

Re: i'm joining the community for contribution

[Ilya Kasnacheev]

Hello!

I have added you to contributors, You should now be able to assign that
issue to yourself.

Regards,
-- 
Ilya Kasnacheev


пн, 8 июн. 2020 г. в 18:43, Aleksey Kurinov :

> Hey, just wanted to add to my previous email
>
> could you please assign
> https://issues.apache.org/jira/browse/IGNITE-13040 task
> to me or advise some another task for beginners
> i can work with SQL, Spring related tasks
>
> -- Forwarded message -
> From: Aleksey Kurinov 
> Date: Mon, Jun 8, 2020 at 8:35 AM
> Subject: i'm joining the community for contribution
> To: 
>
>
> Hello
>
> I'm Aleksei Kurinov,
> Software Developer from Epam Systems company.
> I'd like to contribute to the Ignite project.
> I'm a java developer but never contributed to open source projects.
> I'd like to familiar with the process so for starting I'd prefer to work on
> some trivial task for beginners
> my Jira ID : Kurinov
> email: aleksey.kuri...@gmail.com
>
> thank you
> Aleksey
>

Various shutdown guaranties

[Vladislav Pyatkov]

Hi

We need to have ability to calling shutdown with various guaranties.
For example:
Need to reboot a node, but after that node should be available for
historical rebalance (all partitions in MOVING state should have gone to
OWNING).

Implemented a circled reboot of cluster, but all data should be available
on that time (at least one copy of partition should be available in
cluster).

Need to wait not only data available, but all jobs (before this behavior
available through a stop(false) method invocation).

All these reason required various behavior before shutting down node.
I propose slightly modify public API and add here method which shown on
shutdown behavior directly:
Ignite.close(Shutdown)

public enum Shutdownn {
/**
 * Stop immediately as soon components are ready.
 */
IMMEDIATE,
/**
 * Stop node when all partitions completed moving from/to this node to
another.
 */
NORMAL,
/**
 * Node will stop if and only if it does not store any unique
partitions, that does not have copies on cluster.
 */
GRACEFUL,
/**
 * Node stops graceful and wait all jobs before shutdown.
 */
ALL
}

Method close without parameter Ignite.close() will get shutdown behavior
configured for cluster wide. It will be implemented through distributed
meta storage and additional utilities for configuration.
Also, will be added a method to configure shutdown on start, this is look
as IgniteConfiguration.setShutdown(Shutdown).
If shutting down did not configure all be worked as before according to
IMMEDIATE behavior.
All other close method will be marked as deprecated.

I will be waiting for your opinions.

-- 
Vladislav Pyatkov

Various shutdown guaranties

[V.Pyatkov]

Hi

We need to have ability to calling shutdown with various guaranties.
For example:
Need to reboot a node, but after that node should be available for
historical rebalance (all partitions in MOVING state should have gone to
OWNING).

Implemented a circled reboot of cluster, but all data should be available on
that time (at least one copy of partition should be available in cluster).

Need to wait not only data available, but all jobs (before this behavior
available through a stop(false) method invocation).

All these reason required various behavior before shutting down node.
I propose slightly modify public API and add here method which shown on
shutdown behavior directly:
Ignite.close(Shutdown)

/public enum Shutdownn {
    /**
     * Stop immediately as soon components are ready.
     */
    IMMEDIATE,
    /**
     * Stop node when all partitions completed moving from/to this node to
another.
     */
    NORMAL,
    /**
     * Node will stop if and only if it does not store any unique
partitions, that does not have copies on cluster.
     */
    GRACEFUL,
    /**
     * Node stops graceful and wait all jobs before shutdown.
     */
    ALL
}/

Method close without parameter Ignite.close() will get shutdown behavior
configured for cluster wide. It will be implemented through distributed meta
storage and additional utilities for configuration.
Also, will be added a method to configure shutdown on start, this is look as
IgniteConfiguration.setShutdown(Shutdown).
If shutting down did not configure all be worked as before according to
IMMEDIATE behavior.
All other close method will be marked as deprecated.

I will be waiting for your opinions.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Various shutdown guaranties

[V.Pyatkov]

Hi

We need to have ability to calling shutdown with various guaranties.
For example:
Need to reboot a node, but after that node should be available for
historical rebalance (all partitions in MOVING state should have gone to
OWNING).

Implemented a circled reboot of cluster, but all data should be available on
that time (at least one copy of partition should be available in cluster).

Need to wait not only data available, but all jobs (before this behavior
available through a stop(false) method invocation).

All these reason required various behavior before shutting down node.
I propose slightly modify public API and add here method which shown on
shutdown behavior directly:
Ignite.close(Shutdown)

/public enum Shutdownn {
    /**
     * Stop immediately as soon components are ready.
     */
    IMMEDIATE,
    /**
     * Stop node when all partitions completed moving from/to this node to
another.
     */
    NORMAL,
    /**
     * Node will stop if and only if it does not store any unique
partitions, that does not have copies on cluster.
     */
    GRACEFUL,
    /**
     * Node stops graceful and wait all jobs before shutdown.
     */
    ALL
}/

Method close without parameter Ignite.close() will get shutdown behavior
configured for cluster wide. It will be implemented through distributed meta
storage and additional utilities for configuration.
Also, will be added a method to configure shutdown on start, this is look as
IgniteConfiguration.setShutdown(Shutdown).
If shutting down did not configure all be worked as before according to
IMMEDIATE behavior.
All other close method will be marked as deprecated.

I will be waiting for your opinions.




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Various shutdown guaranties

[V.Pyatkov]

Hi

We need to have ability to calling shutdown with various guaranties.
For example:
Need to reboot a node, but after that node should be available for
historical rebalance (all partitions in MOVING state should have gone to
OWNING).

Implemented a circled reboot of cluster, but all data should be available on
that time (at least one copy of partition should be available in cluster).

Need to wait not only data available, but all jobs (before this behavior
available through a stop(false) method invocation).

All these reason required various behavior before shutting down node.
I propose slightly modify public API and add here method which shown on
shutdown behavior directly:
Ignite.close(Shutdown)

/public enum Shutdownn {
    /**
     * Stop immediately as soon components are ready.
     */
    IMMEDIATE,
    /**
     * Stop node when all partitions completed moving from/to this node to
another.
     */
    NORMAL,
    /**
     * Node will stop if and only if it does not store any unique
partitions, that does not have copies on cluster.
     */
    GRACEFUL,
    /**
     * Node stops graceful and wait all jobs before shutdown.
     */
    ALL
}/

Method close without parameter Ignite.close() will get shutdown behavior
configured for cluster wide. It will be implemented through distributed meta
storage and additional utilities for configuration.
Also, will be added a method to configure shutdown on start, this is look as
IgniteConfiguration.setShutdown(Shutdown).
If shutting down did not configure all be worked as before according to
IMMEDIATE behavior.
All other close method will be marked as deprecated.

I will be waiting for your opinions.




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Fwd: i'm joining the community for contribution

[Aleksey Kurinov]

Hey, just wanted to add to my previous email

could you please assign
https://issues.apache.org/jira/browse/IGNITE-13040 task
to me or advise some another task for beginners
i can work with SQL, Spring related tasks

-- Forwarded message -
From: Aleksey Kurinov 
Date: Mon, Jun 8, 2020 at 8:35 AM
Subject: i'm joining the community for contribution
To: 


Hello

I'm Aleksei Kurinov,
Software Developer from Epam Systems company.
I'd like to contribute to the Ignite project.
I'm a java developer but never contributed to open source projects.
I'd like to familiar with the process so for starting I'd prefer to work on
some trivial task for beginners
my Jira ID : Kurinov
email: aleksey.kuri...@gmail.com

thank you
Aleksey

i'm joining the community for contribution

[Aleksey Kurinov]

Hello

I'm Aleksei Kurinov,
Software Developer from Epam Systems company.
I'd like to contribute to the Ignite project.
I'm a java developer but never contributed to open source projects.
I'd like to familiar with the process so for starting I'd prefer to work on
some trivial task for beginners
my Jira ID : Kurinov
email: aleksey.kuri...@gmail.com

thank you
Aleksey

Re: [DISCUSSION] Separate code sanity check and build task

[Maxim Muzafarov]

Andrey,

I see the following options for PR checks (without intention to merge it):
1. Disable the checkstyle plugin in pom.xml or remove all rules from
checkstyle.xml (for PR branch).
2.
- import checkstyle.xml cofiguration to IntilliJ IDEA inspections
(Preferences > Editor > Inspections)
- be sure that checkbox is switched on for "Checkstyle real-time scan"
- press Ctrl + Alt + L prior to pushing changes to remote (you can
also enable this by default prior to each push)
- PROFIT?!

In general, I don't see any problems at all. What can be simpler of
pressing Crtl + Alt + L prior to the push?

P.S. Sorry for sending an uncomplete e-mail.

On Mon, 8 Jun 2020 at 18:23, Maxim Muzafarov  wrote:
>
> Andrey,
>
> I see the following options for PR checks (without intention to merge it):
> 1.
>
> On Mon, 8 Jun 2020 at 17:59, Andrey Mashenkov
>  wrote:
> >
> > HI,
> >
> > > Why do you want to run a reproducer from the user list on the TC?
> > It may be useful to verify a race that can't be reproduced locally on my NB
> > due to some performance reasons.
> >
> > > If all code style rules are good then we should use them in daily coding,
> > isn’t it?
> > Yes, also we do not force users to rewrite their code regarding our own
> > code style.
> > But I found the code-style (checkstyle, license and inspections) rules
> > force us to do unnecessary work in this particular case,
> > thus makes contribution a bit harder.
> >
> >
> > On Mon, Jun 8, 2020 at 5:27 PM Nikolay Izhikov  wrote:
> >
> > > Hello, Andrey.
> > >
> > > Sorry, I still don’t understand your use-case.
> > > Can you, please, use correct quoting so I can parse your message? :)
> > >
> > > Why do you want to run a reproducer from the user list on the TC?
> > >
> > > > There is no specific rule.
> > >
> > > If all code style rules are good then we should use them in daily coding,
> > > isn’t it?
> > >
> > > Anyway, I think forcing code style rules as early as we can is a good
> > > thing.
> > > Some open-source projects(Kafka, for example) forces code style rules even
> > > on a local test run.
> > >
> > > > 8 июня 2020 г., в 17:22, Andrey Mashenkov 
> > > написал(а):
> > > >
> > > > Hi Niolay,
> > > >
> > > > Why do you ignore code style rules while developing «quick reproducer»?
> > > >
> > > > I don't think it worth effor to fix a user code (e.g. from userlist) 
> > > > just
> > > > to validate some race condition.
> > > >
> > > > There is no specific rule.
> > > > I'm sad we have no ability to run user code without using any hacks to
> > > skip
> > > > style checks with no intention to merge such code to master.
> > > >
> > > >
> > > > On Mon, Jun 8, 2020 at 4:50 PM Nikolay Izhikov 
> > > wrote:
> > > >
> > > >> Hello, Andrey.
> > > >>
> > > >>> I've just found that I should waste my time fixing styles in user code
> > > >> that (may be) reproduce some bug, just to validate the bug or fix
> > > provided
> > > >> by the user.
> > > >>
> > > >> Why do you ignore code style rules while developing «quick reproducer»?
> > > >>
> > > >> What specific code style rule is an issue for you?
> > > >> If we have some rules is just a waste of the time - may be it better to
> > > >> remove them?
> > > >>
> > > >> I’m ++1 to fail the build on code style errors.
> > > >> Code style errors == compile errors for me.
> > > >>
> > > >>> 8 июня 2020 г., в 16:40, Andrey Mashenkov 
> > > >> написал(а):
> > > >>>
> > > >>> Konstantin,
> > > >>>
> > > >>> +1
> > > >>>
> > > >>> I've just found that I should waste my time fixing styles in user code
> > > >> that
> > > >>> (may be) reproduce some bug, just to validate the bug or fix provided
> > > by
> > > >>> the user.
> > > >>> I'm ok with the idea to block commits with style errors to master
> > > branch,
> > > >>> but not for other branches\PR.
> > > >>>
> > > >>> Can anybody explain why commiters should waste their time for this?
> > > >>> Why we even have such a rule to fail build on TC if there is some kind
> > > of
> > > >>> style error?
> > > >>> It looks (like a bullshit) counterintuitive as it is still possible to
> > > >>> commit to master with having style error, but impossible to just build
> > > a
> > > >>> project or to run a test.
> > > >>>
> > > >>>
> > > >>> On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov  > > >
> > > >>> wrote:
> > > >>>
> > >  Hi Ivan,
> > > 
> > >  Thanks for your reply! It’s better to get an answer late than never 
> > >  :)
> > > 
> > >  --
> > >  Regards,
> > >  Konstantin Orlov
> > > 
> > > 
> > > > On 18 May 2020, at 09:04, Ivan Pavlukhin 
> > > wrote:
> > > >
> > > > Hi Konstantin,
> > > >
> > > > Surprisingly, I found your message in a Spam folder (gmail).
> > > >
> > > > We had discussions about the subject before. The most recent one and
> > > > reflecting a current state is [1]. You can find many thoughts and
> > > > arguments in another discussion [2] (it might be better to start
> > > > reading from 

Re: [DISCUSSION] Separate code sanity check and build task

[Maxim Muzafarov]

Andrey,

I see the following options for PR checks (without intention to merge it):
1.

On Mon, 8 Jun 2020 at 17:59, Andrey Mashenkov
 wrote:
>
> HI,
>
> > Why do you want to run a reproducer from the user list on the TC?
> It may be useful to verify a race that can't be reproduced locally on my NB
> due to some performance reasons.
>
> > If all code style rules are good then we should use them in daily coding,
> isn’t it?
> Yes, also we do not force users to rewrite their code regarding our own
> code style.
> But I found the code-style (checkstyle, license and inspections) rules
> force us to do unnecessary work in this particular case,
> thus makes contribution a bit harder.
>
>
> On Mon, Jun 8, 2020 at 5:27 PM Nikolay Izhikov  wrote:
>
> > Hello, Andrey.
> >
> > Sorry, I still don’t understand your use-case.
> > Can you, please, use correct quoting so I can parse your message? :)
> >
> > Why do you want to run a reproducer from the user list on the TC?
> >
> > > There is no specific rule.
> >
> > If all code style rules are good then we should use them in daily coding,
> > isn’t it?
> >
> > Anyway, I think forcing code style rules as early as we can is a good
> > thing.
> > Some open-source projects(Kafka, for example) forces code style rules even
> > on a local test run.
> >
> > > 8 июня 2020 г., в 17:22, Andrey Mashenkov 
> > написал(а):
> > >
> > > Hi Niolay,
> > >
> > > Why do you ignore code style rules while developing «quick reproducer»?
> > >
> > > I don't think it worth effor to fix a user code (e.g. from userlist) just
> > > to validate some race condition.
> > >
> > > There is no specific rule.
> > > I'm sad we have no ability to run user code without using any hacks to
> > skip
> > > style checks with no intention to merge such code to master.
> > >
> > >
> > > On Mon, Jun 8, 2020 at 4:50 PM Nikolay Izhikov 
> > wrote:
> > >
> > >> Hello, Andrey.
> > >>
> > >>> I've just found that I should waste my time fixing styles in user code
> > >> that (may be) reproduce some bug, just to validate the bug or fix
> > provided
> > >> by the user.
> > >>
> > >> Why do you ignore code style rules while developing «quick reproducer»?
> > >>
> > >> What specific code style rule is an issue for you?
> > >> If we have some rules is just a waste of the time - may be it better to
> > >> remove them?
> > >>
> > >> I’m ++1 to fail the build on code style errors.
> > >> Code style errors == compile errors for me.
> > >>
> > >>> 8 июня 2020 г., в 16:40, Andrey Mashenkov 
> > >> написал(а):
> > >>>
> > >>> Konstantin,
> > >>>
> > >>> +1
> > >>>
> > >>> I've just found that I should waste my time fixing styles in user code
> > >> that
> > >>> (may be) reproduce some bug, just to validate the bug or fix provided
> > by
> > >>> the user.
> > >>> I'm ok with the idea to block commits with style errors to master
> > branch,
> > >>> but not for other branches\PR.
> > >>>
> > >>> Can anybody explain why commiters should waste their time for this?
> > >>> Why we even have such a rule to fail build on TC if there is some kind
> > of
> > >>> style error?
> > >>> It looks (like a bullshit) counterintuitive as it is still possible to
> > >>> commit to master with having style error, but impossible to just build
> > a
> > >>> project or to run a test.
> > >>>
> > >>>
> > >>> On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov  > >
> > >>> wrote:
> > >>>
> >  Hi Ivan,
> > 
> >  Thanks for your reply! It’s better to get an answer late than never :)
> > 
> >  --
> >  Regards,
> >  Konstantin Orlov
> > 
> > 
> > > On 18 May 2020, at 09:04, Ivan Pavlukhin 
> > wrote:
> > >
> > > Hi Konstantin,
> > >
> > > Surprisingly, I found your message in a Spam folder (gmail).
> > >
> > > We had discussions about the subject before. The most recent one and
> > > reflecting a current state is [1]. You can find many thoughts and
> > > arguments in another discussion [2] (it might be better to start
> > > reading from a bottom).
> > >
> > > [1]
> > 
> > >>
> > https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
> > > [2]
> > 
> > >>
> > http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
> > >
> > > 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
> > >> Igniters,
> > >>
> > >> Currently we have code sanity checks [1][2] integrated within a
> > build
> >  task
> > >> [3]. Do we really need to fail the build (and therefore the other
> >  tasks) if
> > >> there is a minor flaw like a missing line at the end of a file or an
> >  unused
> > >> import? As for me it could be separated from the build task.
> > >>
> > >> What do you think?
> > >>
> > >> [1]
> > >>
> > 
> > >>
> > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> > >> <
> > 
> > 

Re: [DISCUSSION] Separate code sanity check and build task

[Andrey Mashenkov]

HI,

> Why do you want to run a reproducer from the user list on the TC?
It may be useful to verify a race that can't be reproduced locally on my NB
due to some performance reasons.

> If all code style rules are good then we should use them in daily coding,
isn’t it?
Yes, also we do not force users to rewrite their code regarding our own
code style.
But I found the code-style (checkstyle, license and inspections) rules
force us to do unnecessary work in this particular case,
thus makes contribution a bit harder.


On Mon, Jun 8, 2020 at 5:27 PM Nikolay Izhikov  wrote:

> Hello, Andrey.
>
> Sorry, I still don’t understand your use-case.
> Can you, please, use correct quoting so I can parse your message? :)
>
> Why do you want to run a reproducer from the user list on the TC?
>
> > There is no specific rule.
>
> If all code style rules are good then we should use them in daily coding,
> isn’t it?
>
> Anyway, I think forcing code style rules as early as we can is a good
> thing.
> Some open-source projects(Kafka, for example) forces code style rules even
> on a local test run.
>
> > 8 июня 2020 г., в 17:22, Andrey Mashenkov 
> написал(а):
> >
> > Hi Niolay,
> >
> > Why do you ignore code style rules while developing «quick reproducer»?
> >
> > I don't think it worth effor to fix a user code (e.g. from userlist) just
> > to validate some race condition.
> >
> > There is no specific rule.
> > I'm sad we have no ability to run user code without using any hacks to
> skip
> > style checks with no intention to merge such code to master.
> >
> >
> > On Mon, Jun 8, 2020 at 4:50 PM Nikolay Izhikov 
> wrote:
> >
> >> Hello, Andrey.
> >>
> >>> I've just found that I should waste my time fixing styles in user code
> >> that (may be) reproduce some bug, just to validate the bug or fix
> provided
> >> by the user.
> >>
> >> Why do you ignore code style rules while developing «quick reproducer»?
> >>
> >> What specific code style rule is an issue for you?
> >> If we have some rules is just a waste of the time - may be it better to
> >> remove them?
> >>
> >> I’m ++1 to fail the build on code style errors.
> >> Code style errors == compile errors for me.
> >>
> >>> 8 июня 2020 г., в 16:40, Andrey Mashenkov 
> >> написал(а):
> >>>
> >>> Konstantin,
> >>>
> >>> +1
> >>>
> >>> I've just found that I should waste my time fixing styles in user code
> >> that
> >>> (may be) reproduce some bug, just to validate the bug or fix provided
> by
> >>> the user.
> >>> I'm ok with the idea to block commits with style errors to master
> branch,
> >>> but not for other branches\PR.
> >>>
> >>> Can anybody explain why commiters should waste their time for this?
> >>> Why we even have such a rule to fail build on TC if there is some kind
> of
> >>> style error?
> >>> It looks (like a bullshit) counterintuitive as it is still possible to
> >>> commit to master with having style error, but impossible to just build
> a
> >>> project or to run a test.
> >>>
> >>>
> >>> On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov  >
> >>> wrote:
> >>>
>  Hi Ivan,
> 
>  Thanks for your reply! It’s better to get an answer late than never :)
> 
>  --
>  Regards,
>  Konstantin Orlov
> 
> 
> > On 18 May 2020, at 09:04, Ivan Pavlukhin 
> wrote:
> >
> > Hi Konstantin,
> >
> > Surprisingly, I found your message in a Spam folder (gmail).
> >
> > We had discussions about the subject before. The most recent one and
> > reflecting a current state is [1]. You can find many thoughts and
> > arguments in another discussion [2] (it might be better to start
> > reading from a bottom).
> >
> > [1]
> 
> >>
> https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
> > [2]
> 
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
> >
> > 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
> >> Igniters,
> >>
> >> Currently we have code sanity checks [1][2] integrated within a
> build
>  task
> >> [3]. Do we really need to fail the build (and therefore the other
>  tasks) if
> >> there is a minor flaw like a missing line at the end of a file or an
>  unused
> >> import? As for me it could be separated from the build task.
> >>
> >> What do you think?
> >>
> >> [1]
> >>
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> >> <
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> >
> >> [2]
> >>
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
> >> <
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
> >
> >> [3]
> >>
> 
> >>
> 

Discussion: Existence of manual pinger for ZK client

[Ivan Daschinsky]

Igniters, I suppose, that existence of manual pinger, introduced in [1], in
our ZkDiscoveryImpl is under a big question. This weird thing was
introduced to solve specific issue, specifically strange freeze of all
threads of process during deallocate (Unsafe.freeMemory) of large amount of
data.
Actually, fix [2] introduced workaround for this problem.  Existence of
this "pinger" is not only weird, but actually a little bit ridiculous, even
more this is absolutely nonsense. See [3] for example.

We should remove this strange thing.

[1] - https://issues.apache.org/jira/browse/IGNITE-9683
[2] - https://issues.apache.org/jira/browse/IGNITE-9658
[3] - https://www.usenix.org/legacy/event/atc10/tech/full_papers/Hunt.pdf


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Separate code sanity check and build task

[Nikolay Izhikov]

Hello, Andrey.

Sorry, I still don’t understand your use-case.
Can you, please, use correct quoting so I can parse your message? :)

Why do you want to run a reproducer from the user list on the TC?

> There is no specific rule.

If all code style rules are good then we should use them in daily coding, isn’t 
it?

Anyway, I think forcing code style rules as early as we can is a good thing.
Some open-source projects(Kafka, for example) forces code style rules even on a 
local test run.

> 8 июня 2020 г., в 17:22, Andrey Mashenkov  
> написал(а):
> 
> Hi Niolay,
> 
> Why do you ignore code style rules while developing «quick reproducer»?
> 
> I don't think it worth effor to fix a user code (e.g. from userlist) just
> to validate some race condition.
> 
> There is no specific rule.
> I'm sad we have no ability to run user code without using any hacks to skip
> style checks with no intention to merge such code to master.
> 
> 
> On Mon, Jun 8, 2020 at 4:50 PM Nikolay Izhikov  wrote:
> 
>> Hello, Andrey.
>> 
>>> I've just found that I should waste my time fixing styles in user code
>> that (may be) reproduce some bug, just to validate the bug or fix provided
>> by the user.
>> 
>> Why do you ignore code style rules while developing «quick reproducer»?
>> 
>> What specific code style rule is an issue for you?
>> If we have some rules is just a waste of the time - may be it better to
>> remove them?
>> 
>> I’m ++1 to fail the build on code style errors.
>> Code style errors == compile errors for me.
>> 
>>> 8 июня 2020 г., в 16:40, Andrey Mashenkov 
>> написал(а):
>>> 
>>> Konstantin,
>>> 
>>> +1
>>> 
>>> I've just found that I should waste my time fixing styles in user code
>> that
>>> (may be) reproduce some bug, just to validate the bug or fix provided by
>>> the user.
>>> I'm ok with the idea to block commits with style errors to master branch,
>>> but not for other branches\PR.
>>> 
>>> Can anybody explain why commiters should waste their time for this?
>>> Why we even have such a rule to fail build on TC if there is some kind of
>>> style error?
>>> It looks (like a bullshit) counterintuitive as it is still possible to
>>> commit to master with having style error, but impossible to just build a
>>> project or to run a test.
>>> 
>>> 
>>> On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov 
>>> wrote:
>>> 
 Hi Ivan,
 
 Thanks for your reply! It’s better to get an answer late than never :)
 
 --
 Regards,
 Konstantin Orlov
 
 
> On 18 May 2020, at 09:04, Ivan Pavlukhin  wrote:
> 
> Hi Konstantin,
> 
> Surprisingly, I found your message in a Spam folder (gmail).
> 
> We had discussions about the subject before. The most recent one and
> reflecting a current state is [1]. You can find many thoughts and
> arguments in another discussion [2] (it might be better to start
> reading from a bottom).
> 
> [1]
 
>> https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
> [2]
 
>> http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
> 
> 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
>> Igniters,
>> 
>> Currently we have code sanity checks [1][2] integrated within a build
 task
>> [3]. Do we really need to fail the build (and therefore the other
 tasks) if
>> there is a minor flaw like a missing line at the end of a file or an
 unused
>> import? As for me it could be separated from the build task.
>> 
>> What do you think?
>> 
>> [1]
>> 
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
>> <
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> 
>> [2]
>> 
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
>> <
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
> 
>> [3]
>> 
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
>> <
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
> 
>> 
>> --
>> Regards,
>> Konstantin Orlov
>> 
>> 
>> 
> 
> 
> --
> 
> Best regards,
> Ivan Pavlukhin
 
 
>>> 
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>> 
>> 
> 
> -- 
> Best regards,
> Andrey V. Mashenkov

Re: [DISCUSSION] Separate code sanity check and build task

[Andrey Mashenkov]

Hi Niolay,

Why do you ignore code style rules while developing «quick reproducer»?

I don't think it worth effor to fix a user code (e.g. from userlist) just
to validate some race condition.

There is no specific rule.
I'm sad we have no ability to run user code without using any hacks to skip
style checks with no intention to merge such code to master.


On Mon, Jun 8, 2020 at 4:50 PM Nikolay Izhikov  wrote:

> Hello, Andrey.
>
> > I've just found that I should waste my time fixing styles in user code
> that (may be) reproduce some bug, just to validate the bug or fix provided
> by the user.
>
> Why do you ignore code style rules while developing «quick reproducer»?
>
> What specific code style rule is an issue for you?
> If we have some rules is just a waste of the time - may be it better to
> remove them?
>
> I’m ++1 to fail the build on code style errors.
> Code style errors == compile errors for me.
>
> > 8 июня 2020 г., в 16:40, Andrey Mashenkov 
> написал(а):
> >
> > Konstantin,
> >
> > +1
> >
> > I've just found that I should waste my time fixing styles in user code
> that
> > (may be) reproduce some bug, just to validate the bug or fix provided by
> > the user.
> > I'm ok with the idea to block commits with style errors to master branch,
> > but not for other branches\PR.
> >
> > Can anybody explain why commiters should waste their time for this?
> > Why we even have such a rule to fail build on TC if there is some kind of
> > style error?
> > It looks (like a bullshit) counterintuitive as it is still possible to
> > commit to master with having style error, but impossible to just build a
> > project or to run a test.
> >
> >
> > On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov 
> > wrote:
> >
> >> Hi Ivan,
> >>
> >> Thanks for your reply! It’s better to get an answer late than never :)
> >>
> >> --
> >> Regards,
> >> Konstantin Orlov
> >>
> >>
> >>> On 18 May 2020, at 09:04, Ivan Pavlukhin  wrote:
> >>>
> >>> Hi Konstantin,
> >>>
> >>> Surprisingly, I found your message in a Spam folder (gmail).
> >>>
> >>> We had discussions about the subject before. The most recent one and
> >>> reflecting a current state is [1]. You can find many thoughts and
> >>> arguments in another discussion [2] (it might be better to start
> >>> reading from a bottom).
> >>>
> >>> [1]
> >>
> https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
> >>> [2]
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
> >>>
> >>> 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
>  Igniters,
> 
>  Currently we have code sanity checks [1][2] integrated within a build
> >> task
>  [3]. Do we really need to fail the build (and therefore the other
> >> tasks) if
>  there is a minor flaw like a missing line at the end of a file or an
> >> unused
>  import? As for me it could be separated from the build task.
> 
>  What do you think?
> 
>  [1]
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
>  <
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> >>>
>  [2]
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
>  <
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
> >>>
>  [3]
> 
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
>  <
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
> >>>
> 
>  --
>  Regards,
>  Konstantin Orlov
> 
> 
> 
> >>>
> >>>
> >>> --
> >>>
> >>> Best regards,
> >>> Ivan Pavlukhin
> >>
> >>
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>

-- 
Best regards,
Andrey V. Mashenkov

Re: [DISCUSSION] Separate code sanity check and build task

[Nikolay Izhikov]

Hello, Andrey.

> I've just found that I should waste my time fixing styles in user code that 
> (may be) reproduce some bug, just to validate the bug or fix provided by the 
> user.

Why do you ignore code style rules while developing «quick reproducer»?

What specific code style rule is an issue for you?
If we have some rules is just a waste of the time - may be it better to remove 
them?

I’m ++1 to fail the build on code style errors.
Code style errors == compile errors for me.

> 8 июня 2020 г., в 16:40, Andrey Mashenkov  
> написал(а):
> 
> Konstantin,
> 
> +1
> 
> I've just found that I should waste my time fixing styles in user code that
> (may be) reproduce some bug, just to validate the bug or fix provided by
> the user.
> I'm ok with the idea to block commits with style errors to master branch,
> but not for other branches\PR.
> 
> Can anybody explain why commiters should waste their time for this?
> Why we even have such a rule to fail build on TC if there is some kind of
> style error?
> It looks (like a bullshit) counterintuitive as it is still possible to
> commit to master with having style error, but impossible to just build a
> project or to run a test.
> 
> 
> On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov 
> wrote:
> 
>> Hi Ivan,
>> 
>> Thanks for your reply! It’s better to get an answer late than never :)
>> 
>> --
>> Regards,
>> Konstantin Orlov
>> 
>> 
>>> On 18 May 2020, at 09:04, Ivan Pavlukhin  wrote:
>>> 
>>> Hi Konstantin,
>>> 
>>> Surprisingly, I found your message in a Spam folder (gmail).
>>> 
>>> We had discussions about the subject before. The most recent one and
>>> reflecting a current state is [1]. You can find many thoughts and
>>> arguments in another discussion [2] (it might be better to start
>>> reading from a bottom).
>>> 
>>> [1]
>> https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
>>> [2]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
>>> 
>>> 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
 Igniters,
 
 Currently we have code sanity checks [1][2] integrated within a build
>> task
 [3]. Do we really need to fail the build (and therefore the other
>> tasks) if
 there is a minor flaw like a missing line at the end of a file or an
>> unused
 import? As for me it could be separated from the build task.
 
 What do you think?
 
 [1]
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
 <
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
>>> 
 [2]
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
 <
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
>>> 
 [3]
 
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
 <
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
>>> 
 
 --
 Regards,
 Konstantin Orlov
 
 
 
>>> 
>>> 
>>> --
>>> 
>>> Best regards,
>>> Ivan Pavlukhin
>> 
>> 
> 
> -- 
> Best regards,
> Andrey V. Mashenkov

Re: [DISCUSSION] Separate code sanity check and build task

[Andrey Mashenkov]

Konstantin,

+1

I've just found that I should waste my time fixing styles in user code that
(may be) reproduce some bug, just to validate the bug or fix provided by
the user.
I'm ok with the idea to block commits with style errors to master branch,
but not for other branches\PR.

Can anybody explain why commiters should waste their time for this?
Why we even have such a rule to fail build on TC if there is some kind of
style error?
It looks (like a bullshit) counterintuitive as it is still possible to
commit to master with having style error, but impossible to just build a
project or to run a test.


On Thu, May 21, 2020 at 12:23 PM Konstantin Orlov 
wrote:

> Hi Ivan,
>
> Thanks for your reply! It’s better to get an answer late than never :)
>
> --
> Regards,
> Konstantin Orlov
>
>
> > On 18 May 2020, at 09:04, Ivan Pavlukhin  wrote:
> >
> > Hi Konstantin,
> >
> > Surprisingly, I found your message in a Spam folder (gmail).
> >
> > We had discussions about the subject before. The most recent one and
> > reflecting a current state is [1]. You can find many thoughts and
> > arguments in another discussion [2] (it might be better to start
> > reading from a bottom).
> >
> > [1]
> https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
> > [2]
> http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
> >
> > 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
> >> Igniters,
> >>
> >> Currently we have code sanity checks [1][2] integrated within a build
> task
> >> [3]. Do we really need to fail the build (and therefore the other
> tasks) if
> >> there is a minor flaw like a missing line at the end of a file or an
> unused
> >> import? As for me it could be separated from the build task.
> >>
> >> What do you think?
> >>
> >> [1]
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> >> <
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
> >
> >> [2]
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
> >> <
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
> >
> >> [3]
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
> >> <
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
> >
> >>
> >> --
> >> Regards,
> >> Konstantin Orlov
> >>
> >>
> >>
> >
> >
> > --
> >
> > Best regards,
> > Ivan Pavlukhin
>
>

-- 
Best regards,
Andrey V. Mashenkov

Re: Various shutdown guaranties

[V.Pyatkov]

> I see no relation between possible data loss on shutdown and waiting for
> some jobs to complete.

Yes, I do not think that is.
More over that I want to union waiting of jobs and waiting rebalance in one
shutdown policy. Rather, these various shutdown types mean various types of
waiting.
IMMEDIATE - nothing to wait, simply calling stop on components.
NORMAL - wait all what we can on local node before calling close.
GRACEFUL - More strict, in additional we will wait some distributed
conditions (for example waiting of nodes with copies of local partitions).

No more over that. I will prefer exclude ALL.
enum Shutdownn {
IMMEDIATE,
NORMAL,
GRACEFUL
}

> IgniteConfiguration.setGracefulShutdownTimeout(long);  // Force a shutdown

I suppose that it is available manually. If client consider that node is
stopping is very long, it can to invoke Ignite.close(IMMEDIATE) (or kill -9
).

> IgniteConfiguration.setShutdownPolicy(GRACEFUL|DEFAULT);

That does not allow deciding shutdown policy in runtime. If it normally we
can to hide method stop(Shutdown) in internal API.

> The properties also should be changeable at runtime using JMX or
> control.sh interface.

Before I was sure the shutdown type will be stored in cluster, but our
implicit policy says: "All properties which configured through JMX will not
save after restart."




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

IGNITE-12808 Allow create tables for existing caches

[Ivan Daschinsky]

Hi Igniters.

I've recently submitted patch[1] that implemented subj.
Any help with review would be appreciated.


[1] - https://issues.apache.org/jira/browse/IGNITE-12808
-- 
Sincerely yours, Ivan Daschinskiy

Re: Help with ignite cluster

[Ilya Kasnacheev]

Hello!

How do you find out that you can only access data from one node but not the
other? How does it manifest?

Regards,
-- 
Ilya Kasnacheev


чт, 4 июн. 2020 г. в 18:26, C Ravikiran :

> I am able to see Topology snapshot in my logs as ver=23 servers=2
> offheap=93 gb heap=120 gb
>
> I am able see 2 severs.
>
> Still we are facing the issue.
>
> Could you please help me on this issue.
>
> On Thu, 4 Jun, 2020, 5:34 pm Ilya Kasnacheev, 
> wrote:
>
> > Hello!
> >
> > I think that your nodes were not able to discover each other and formed
> two
> > clusters.
> >
> > Can you find "Topology snapshot" messages from their logs, share those
> with
> > us?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 4 июн. 2020 г. в 13:28, C Ravikiran :
> >
> > > Hi Team,
> > >
> > > Initially my application running with single node.
> > >
> > > Recently added one more node in to the cluster and my application is
> > > working with two nodes without issues and also fetching data in cache
> IDs
> > > also.
> > >
> > > But today one node was down which was runnig intially.
> > >
> > > Newly added node was running fine.
> > >
> > > But we are not able to see the data from newly added node.
> > >
> > > Two nodes sync has not done properly.
> > >
> > > We are using ignite 2.6 version
> > >
> > > Cache mode: partitioned, FULL_SYNC
> > >
> > > Could you please help me on this.
> > > Ignite ui i am getting below error:
> > > Fail to find data nodes for cache: *
> > >
> > > Thanks in advance
> > > advanc. e
> > >
> >
>

Re: Various shutdown guaranties

[Alexei Scherbakov]

Graceful policy should only be applicable to caches having a number of
backups > 0.

пн, 8 июн. 2020 г. в 14:54, Alexei Scherbakov :

> V.Pyatkov
>
>
> While I agree we need a way to prevent unintentional data loss on
> shutdown, I do not like the proposed shutdown flags enum.
> I see no relation between possible data loss on shutdown and waiting for
> some jobs to complete.
>
> All we need is a new method (duplicated by system property), like
>
> IgniteConfiguration.setShutdownPolicy(GRACEFUL|DEFAULT);
> and an optional
> IgniteConfiguration.setGracefulShutdownTimeout(long);  // Force a shutdown
> if the timeout is expired.
>
> For enabled graceful policy a node shouldn't normally stop if it is the
> last owner for any partition.
> This will prevent unintentional data loss on stop when it is possible, for
> example if a grid is deployed over kubernetes.
>
> The properties also should be changeable at runtime using JMX or
> control.sh interface.
>
>
>
>
> пн, 8 июн. 2020 г. в 13:46, V.Pyatkov :
>
>> Hi
>>
>> We need to have ability to calling shutdown with various guaranties.
>> For example:
>> Need to reboot a node, but after that node should be available for
>> historical rebalance (all partitions in MOVING state should have gone to
>> OWNING).
>>
>> Implemented a circled reboot of cluster, but all data should be available
>> on
>> that time (at least one copy of partition should be available in cluster).
>>
>> Need to wait not only data available, but all jobs (before this behavior
>> available through a stop(false) method invocation).
>>
>> All these reason required various behavior before shutting down node.
>> I propose slightly modify public API and add here method which shown on
>> shutdown behavior directly:
>> Ignite.close(Shutdown)
>>
>> /public enum Shutdownn {
>> /**
>>  * Stop immediately as soon components are ready.
>>  */
>> IMMEDIATE,
>> /**
>>  * Stop node when all partitions completed moving from/to this node to
>> another.
>>  */
>> NORMAL,
>> /**
>>  * Node will stop if and only if it does not store any unique
>> partitions, that does not have copies on cluster.
>>  */
>> GRACEFUL,
>> /**
>>  * Node stops graceful and wait all jobs before shutdown.
>>  */
>> ALL
>> }/
>>
>> Method close without parameter Ignite.close() will get shutdown behavior
>> configured for cluster wide. It will be implemented through distributed
>> meta
>> storage and additional utilities for configuration.
>> Also, will be added a method to configure shutdown on start, this is look
>> as
>> IgniteConfiguration.setShutdown(Shutdown).
>> If shutting down did not configure all be worked as before according to
>> IMMEDIATE behavior.
>> All other close method will be marked as deprecated.
>>
>> I will be waiting for your opinions.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>


-- 

Best regards,
Alexei Scherbakov

Re: Various shutdown guaranties

[Alexei Scherbakov]

V.Pyatkov


While I agree we need a way to prevent unintentional data loss on shutdown,
I do not like the proposed shutdown flags enum.
I see no relation between possible data loss on shutdown and waiting for
some jobs to complete.

All we need is a new method (duplicated by system property), like

IgniteConfiguration.setShutdownPolicy(GRACEFUL|DEFAULT);
and an optional
IgniteConfiguration.setGracefulShutdownTimeout(long);  // Force a shutdown
if the timeout is expired.

For enabled graceful policy a node shouldn't normally stop if it is the
last owner for any partition.
This will prevent unintentional data loss on stop when it is possible, for
example if a grid is deployed over kubernetes.

The properties also should be changeable at runtime using JMX or control.sh
interface.




пн, 8 июн. 2020 г. в 13:46, V.Pyatkov :

> Hi
>
> We need to have ability to calling shutdown with various guaranties.
> For example:
> Need to reboot a node, but after that node should be available for
> historical rebalance (all partitions in MOVING state should have gone to
> OWNING).
>
> Implemented a circled reboot of cluster, but all data should be available
> on
> that time (at least one copy of partition should be available in cluster).
>
> Need to wait not only data available, but all jobs (before this behavior
> available through a stop(false) method invocation).
>
> All these reason required various behavior before shutting down node.
> I propose slightly modify public API and add here method which shown on
> shutdown behavior directly:
> Ignite.close(Shutdown)
>
> /public enum Shutdownn {
> /**
>  * Stop immediately as soon components are ready.
>  */
> IMMEDIATE,
> /**
>  * Stop node when all partitions completed moving from/to this node to
> another.
>  */
> NORMAL,
> /**
>  * Node will stop if and only if it does not store any unique
> partitions, that does not have copies on cluster.
>  */
> GRACEFUL,
> /**
>  * Node stops graceful and wait all jobs before shutdown.
>  */
> ALL
> }/
>
> Method close without parameter Ignite.close() will get shutdown behavior
> configured for cluster wide. It will be implemented through distributed
> meta
> storage and additional utilities for configuration.
> Also, will be added a method to configure shutdown on start, this is look
> as
> IgniteConfiguration.setShutdown(Shutdown).
> If shutting down did not configure all be worked as before according to
> IMMEDIATE behavior.
> All other close method will be marked as deprecated.
>
> I will be waiting for your opinions.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


-- 

Best regards,
Alexei Scherbakov

Re: Question: network issues of single node.

[Sergey Chugunov]

Of course I meant ticket [1] increased cluster stability in situation of
blinking network.

[1] https://issues.apache.org/jira/browse/IGNITE-7163

On Mon, Jun 8, 2020 at 1:51 PM Sergey Chugunov 
wrote:

> Vladimir,
>
> Adding to what Alexey has said I remember that cases of short-term network
> issues (blinking network) were also a driver for this improvement. They are
> indeed hard to reproduce but have been seen in real world set-ups and have
> proven to increase cluster stability.
>
> On Sat, Jun 6, 2020 at 5:09 PM Denis Magda  wrote:
>
>> Finally, I got your question.
>>
>> Back in 2017-2018, there was a Discovery SPI's stabilization activity. The
>> networking component could fail in various hard-to-reproduce scenarios
>> affecting cluster availability and consistency. That ticket reminds me of
>> those notorious issues that would fire once a week or month under specific
>> configuration settings. So, I would not touch the code that fixes the
>> issue
>> unless @Alexey Goncharuk  or @Sergey Chugunov
>>  confirms that it's safe to do. Also, there
>> should
>> be a test for this scenario.
>>
>> -
>> Denis
>>
>>
>> On Fri, Jun 5, 2020 at 12:28 AM Vladimir Steshin 
>> wrote:
>>
>> > Denis,
>> >
>> > I have no nodes that I'm unable to interconnect. This case is simulated
>> > in IgniteDiscoveryMassiveNodeFailTest.testMassiveFailSelfKill()
>> > Introduced in [1].
>> >
>> > I’m asking if it is real or supposed problem. Where it was met? Which
>> > network configuration/issues could be?
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-7163
>> >
>> > 05.06.2020 1:01, Denis Magda пишет:
>> > > Vladimir,
>> > >
>> > > I'm suggesting to share the log files from the nodes that are unable
>> to
>> > > interconnect so that the community can check them for potential
>> issues.
>> > > Instead of sharing the logs from all the 5 nodes, try to start a
>> > two-nodes
>> > > cluster with the nodes that fail to discover each other and attach the
>> > logs
>> > > from those.
>> > >
>> > > -
>> > > Denis
>> > >
>> > >
>> > > On Thu, Jun 4, 2020 at 1:57 PM Vladimir Steshin 
>> > wrote:
>> > >
>> > >> Denis, hi.
>> > >>
>> > >>   Sorry, I didn’t catch your idea. Are you saying this can happen
>> > and
>> > >> suggest experiment? I’m not descripting a probable case. It is
>> already
>> > >> done in [1]. I’m asking is it real, where it was met.
>> > >>
>> > >>
>> > >> 04.06.2020 23:33, Denis Magda пишет:
>> > >>> Vladimir,
>> > >>>
>> > >>> Please do the following experiment. Start a 2-nodes cluster booting
>> > node
>> > >> 3
>> > >>> and, for instance, node 5. Those won't be able to interconnect
>> > according
>> > >> to
>> > >>> your description. Attach the log files from both nodes for analysis.
>> > This
>> > >>> should be a networking issue.
>> > >>>
>> > >>> -
>> > >>> Denis
>> > >>>
>> > >>>
>> > >>> On Thu, Jun 4, 2020 at 1:24 PM Vladimir Steshin > >
>> > >> wrote:
>> > Hi, Igniters.
>> > 
>> > 
>> > I wanted to ask how one node may not be able to connect to
>> > another
>> >  whereas rest of the cluster can. This got covered in [1]. In short:
>> > node
>> >  3 can't connect to nodes 4 and 5 but can to 1. At the same time,
>> node
>> > 2
>> >  can connect to 4. Questions:
>> > 
>> >  1) Is it real case? Where this problem came from?
>> > 
>> >  2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t
>> > connect
>> >  to 4 (and 5) too?
>> > 
>> >  Sergey, Dmitry maybe you bring light (I see you in [1])? I'm
>> >  participating in [2] and found this backward connection checking.
>> >  Answering would help us a lot.
>> > 
>> >  Thanks!
>> > 
>> >  [1]
>> >  https://issues.apache.org/jira/browse/IGNITE-7163<
>> >  https://issues.apache.org/jira/browse/IGNITE-7163>
>> > 
>> >  [2]
>> > 
>> > 
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
>> >  <
>> > 
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
>> >
>>
>

Re: Question: network issues of single node.

[Sergey Chugunov]

Vladimir,

Adding to what Alexey has said I remember that cases of short-term network
issues (blinking network) were also a driver for this improvement. They are
indeed hard to reproduce but have been seen in real world set-ups and have
proven to increase cluster stability.

On Sat, Jun 6, 2020 at 5:09 PM Denis Magda  wrote:

> Finally, I got your question.
>
> Back in 2017-2018, there was a Discovery SPI's stabilization activity. The
> networking component could fail in various hard-to-reproduce scenarios
> affecting cluster availability and consistency. That ticket reminds me of
> those notorious issues that would fire once a week or month under specific
> configuration settings. So, I would not touch the code that fixes the issue
> unless @Alexey Goncharuk  or @Sergey Chugunov
>  confirms that it's safe to do. Also, there should
> be a test for this scenario.
>
> -
> Denis
>
>
> On Fri, Jun 5, 2020 at 12:28 AM Vladimir Steshin 
> wrote:
>
> > Denis,
> >
> > I have no nodes that I'm unable to interconnect. This case is simulated
> > in IgniteDiscoveryMassiveNodeFailTest.testMassiveFailSelfKill()
> > Introduced in [1].
> >
> > I’m asking if it is real or supposed problem. Where it was met? Which
> > network configuration/issues could be?
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7163
> >
> > 05.06.2020 1:01, Denis Magda пишет:
> > > Vladimir,
> > >
> > > I'm suggesting to share the log files from the nodes that are unable to
> > > interconnect so that the community can check them for potential issues.
> > > Instead of sharing the logs from all the 5 nodes, try to start a
> > two-nodes
> > > cluster with the nodes that fail to discover each other and attach the
> > logs
> > > from those.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Jun 4, 2020 at 1:57 PM Vladimir Steshin 
> > wrote:
> > >
> > >> Denis, hi.
> > >>
> > >>   Sorry, I didn’t catch your idea. Are you saying this can happen
> > and
> > >> suggest experiment? I’m not descripting a probable case. It is already
> > >> done in [1]. I’m asking is it real, where it was met.
> > >>
> > >>
> > >> 04.06.2020 23:33, Denis Magda пишет:
> > >>> Vladimir,
> > >>>
> > >>> Please do the following experiment. Start a 2-nodes cluster booting
> > node
> > >> 3
> > >>> and, for instance, node 5. Those won't be able to interconnect
> > according
> > >> to
> > >>> your description. Attach the log files from both nodes for analysis.
> > This
> > >>> should be a networking issue.
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Thu, Jun 4, 2020 at 1:24 PM Vladimir Steshin 
> > >> wrote:
> > Hi, Igniters.
> > 
> > 
> > I wanted to ask how one node may not be able to connect to
> > another
> >  whereas rest of the cluster can. This got covered in [1]. In short:
> > node
> >  3 can't connect to nodes 4 and 5 but can to 1. At the same time,
> node
> > 2
> >  can connect to 4. Questions:
> > 
> >  1) Is it real case? Where this problem came from?
> > 
> >  2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t
> > connect
> >  to 4 (and 5) too?
> > 
> >  Sergey, Dmitry maybe you bring light (I see you in [1])? I'm
> >  participating in [2] and found this backward connection checking.
> >  Answering would help us a lot.
> > 
> >  Thanks!
> > 
> >  [1]
> >  https://issues.apache.org/jira/browse/IGNITE-7163<
> >  https://issues.apache.org/jira/browse/IGNITE-7163>
> > 
> >  [2]
> > 
> > 
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >  <
> > 
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >
>

Re: Tool for performance statistics reports

[Nikolay Izhikov]

Hello, Alexey.

Thanks for the review.

My understanding if the following:

We will have 3 in-depth tool to find issues in cluster:

1. Metrics + System views - data that describe Ignite entities very high-level.

2. Profiling - tool to know what specific query of transactions are slow. 
In many cases, this knowledge is enough to fix the issue(rewrite query, 
redesign transactions flow, etc)

3. Tracing - tool to know why one of 1000 of the same queries was slow.
The most detailed view of the Ignite internal processes.

> For example, a user would not be able to match a long task with a long job in 
> that task.

This is not true.
Profiling report will aggregate data from all nodes.
So there will be both

 * summary time of the task
 * time of the each job in the task.


> 8 июня 2020 г., в 12:52, Alexey Goncharuk  
> написал(а):
> 
> Nikita, Igniters,
> 
> I left a few comments on the tool itself in the PR.
> 
> However, I would like to reiterate and discuss why a user would prefer to
> use the profiling tool over tracing? Profiling tool only captures very
> high-level details of the operations (a single cache operation, for
> example), and does not interconnect operations happened on different nodes.
> For example, a user would not be able to match a long task with a long job
> in that task. In other words, profiling data is always a subset of data
> collected by tracing.
> 
> Maybe it makes sense to adopt local log file approach to write spans so we
> can process those spans later to build a report?
> 
> чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev :
> 
>> Hi, Igniters.
>> 
>> I have implemented cluster profiling and tool to build the performance
>> report. It's ready to be reviewed. [1, 2]
>> 
>> Profiling can be managed by JMX bean. I have plans to implement it to
>> control.sh also.
>> 
>> Nodes write statistics to the temporary off heap buffer and then one
>> thread flushes to the profiling files. The write mechanics and format
>> is like WAL logging.
>> The report contains the following statistics:
>> - nodes and caches info
>> - cache operations and transaction statistics
>> - SQL and scan queries statistics (include logical and physical reads per
>> query)
>> - tasks and jobs statistics.
>> 
>> More details in the IEP [3].
>> 
>> [1] https://github.com/apache/ignite/pull/7693
>> [2] https://issues.apache.org/jira/browse/IGNITE-12666
>> [3]
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> 
>> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин :
>>> 
>>> Hello Nikolay,
>>> 
 Who deprecated visor and when? Maybe I miss something?
>>> On the one hand, there was technically no community consensus that this
>>> tool should be obsolete.
>>> On the other hand, my opinion based on the following topic:
>>> 
>> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
>>> Moreover, it seems to me, currently, the control utility is widely used
>> and
>>> actively developed, instead of the visor.
>>> 
 It's true that, for now, Ignite doesn't have "tool strategy" I think
>> it's
>>> a big issue from the user's point of view.
>>> I absolutely agree with that.
>>> 
 We should solve it in the nearest time. Feel free to start this
>> activity
>>> I have no plan at the moment. However, at the first stage, we could
>>> understand the difference between visor and control utility.
>>> All useful features from visor should be moved/implemented in control
>>> utility and after that visor tool and should be marked as
>>> deprecated/obsoleted.
>>> 
 You need to throw in control.sh also, which does some kind of
>> statistics
>>> too, such as idle_verify.
 Please, clarify your idea:
   We should use some info from control.sh to the report?
   The report should be generated by some control.sh subcommand?
>>> If I am not mistaken, the oracle database has AWR tool (mentioned on the
>>> IEP page) which is a command-line utility that generates HTML reports.
>>> I like this idea and I think this is a good approach that can be realized
>>> in the control utility.
>>> If we have a case that cannot be implemented in this way, we have to
>>> clearly states the difference between these tools so as not to confuse
>> our
>>> users.
>>> What do you think?
>>> 
>>> Thanks,
>>> Slava.
>>> 
>>> 
>>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov :
>>> 
 Hello, Slava, Ilya, Denis.
 
 Thanks for joining this discussion!
 
> - visor (which is deprecated)
 
 Who deprecated visor and when?
 Maybe I miss something?
 
> - web-console (to be honest, I don't quite understand the status of
>> this
 tool)
 
 +1.
 
> I am not against the new tool, I just want to understand the
>> motivation
 to not improve the existing sub-projects.
 
 It's true that, for now, Ignite doesn't have "tool strategy"
 I think it's a big issue from the user's point of view.
 We should solve it in the 

Various shutdown guaranties

[V.Pyatkov]

Hi

We need to have ability to calling shutdown with various guaranties.
For example:
Need to reboot a node, but after that node should be available for
historical rebalance (all partitions in MOVING state should have gone to
OWNING).

Implemented a circled reboot of cluster, but all data should be available on
that time (at least one copy of partition should be available in cluster).

Need to wait not only data available, but all jobs (before this behavior
available through a stop(false) method invocation).

All these reason required various behavior before shutting down node.
I propose slightly modify public API and add here method which shown on
shutdown behavior directly:
Ignite.close(Shutdown)

/public enum Shutdownn {
    /**
     * Stop immediately as soon components are ready.
     */
    IMMEDIATE,
    /**
     * Stop node when all partitions completed moving from/to this node to
another.
     */
    NORMAL,
    /**
     * Node will stop if and only if it does not store any unique
partitions, that does not have copies on cluster.
     */
    GRACEFUL,
    /**
     * Node stops graceful and wait all jobs before shutdown.
     */
    ALL
}/

Method close without parameter Ignite.close() will get shutdown behavior
configured for cluster wide. It will be implemented through distributed meta
storage and additional utilities for configuration.
Also, will be added a method to configure shutdown on start, this is look as
IgniteConfiguration.setShutdown(Shutdown).
If shutting down did not configure all be worked as before according to
IMMEDIATE behavior.
All other close method will be marked as deprecated.

I will be waiting for your opinions.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Countdown latch issue with 2.6.0

[Akash Shinde]

Hi, I have created jira for this issue
https://issues.apache.org/jira/browse/IGNITE-13132

Thanks,
Akash

On Sun, Jun 7, 2020 at 9:29 AM Akash Shinde  wrote:

> Can someone please help me with this issue.
>
> On Sat, Jun 6, 2020 at 6:45 PM Akash Shinde  wrote:
>
>> Hi,
>> Issue: Countdown latched gets reinitialize to original value(4) when one
>> or more (but not all) node goes down. (Partition loss happened)
>>
>> We are using ignite's distributed countdownlatch to make sure that cache
>> loading is completed on all server nodes. We do this to make sure that our
>> kafka consumers starts only after cache loading is complete on all server
>> nodes. This is the basic criteria which needs to be fulfilled before starts
>> actual processing
>>
>>
>>  We have 4 server nodes and countdownlatch is initialized to 4. We use
>> "cache.loadCache" method to start the cache loading. When each server
>> completes cache loading it reduces the count by 1 using countDown method.
>> So when all the nodes completes cache loading, the count reaches to zero.
>> When this count  reaches to zero we start kafka consumers on all server
>> nodes.
>>
>>  But we saw weird behavior in prod env. The 3 server nodes were shut down
>> at the same time. But 1 node is still alive. When this happened the count
>> down was reinitialized to original value i.e. 4. But I am not able to
>> reproduce this in dev env.
>>
>>  Is this a bug, when one or more (but not all) nodes goes down then count
>> re initializes back to original value?
>>
>> Thanks,
>> Akash
>>
>

[jira] [Created] (IGNITE-13132) Countdown latched gets reinitialize to original value(4) when one or more (but not all) node goes down.

[Akash Shinde (Jira)]

Akash Shinde created IGNITE-13132:
-

 Summary: Countdown latched gets reinitialize to original value(4) 
when one or more (but not all) node goes down. 
 Key: IGNITE-13132
 URL: https://issues.apache.org/jira/browse/IGNITE-13132
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Reporter: Akash Shinde


We are using ignite's distributed countdownlatch to make sure that cache 
loading is completed on all server nodes. We do this to make sure that our 
kafka consumers starts only after cache loading is complete on all server 
nodes. This is the basic criteria which needs to be fulfilled before starts 
actual processing

 We have 4 server nodes and countdownlatch is initialized to 4. We use 
"cache.loadCache" method to start the cache loading. When each server completes 
cache loading it reduces the count by 1 using countDown method. So when all the 
nodes completes cache loading, the count reaches to zero. When this count  
reaches to zero we start kafka consumers on all server nodes.

 But we saw weird behavior in prod env. The 3 server nodes were shut down at 
the same time. But 1 node is still alive. When this happened the count down was 
reinitialized to original value i.e. 4. But I am not able to reproduce this in 
dev env.  
 
Note: Partiton loss were happened when three node gone down at same time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Tool for performance statistics reports

[Alexey Goncharuk]

Nikita, Igniters,

I left a few comments on the tool itself in the PR.

However, I would like to reiterate and discuss why a user would prefer to
use the profiling tool over tracing? Profiling tool only captures very
high-level details of the operations (a single cache operation, for
example), and does not interconnect operations happened on different nodes.
For example, a user would not be able to match a long task with a long job
in that task. In other words, profiling data is always a subset of data
collected by tracing.

Maybe it makes sense to adopt local log file approach to write spans so we
can process those spans later to build a report?

чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev :

> Hi, Igniters.
>
> I have implemented cluster profiling and tool to build the performance
> report. It's ready to be reviewed. [1, 2]
>
> Profiling can be managed by JMX bean. I have plans to implement it to
> control.sh also.
>
> Nodes write statistics to the temporary off heap buffer and then one
> thread flushes to the profiling files. The write mechanics and format
> is like WAL logging.
> The report contains the following statistics:
> - nodes and caches info
> - cache operations and transaction statistics
> - SQL and scan queries statistics (include logical and physical reads per
> query)
> - tasks and jobs statistics.
>
> More details in the IEP [3].
>
> [1] https://github.com/apache/ignite/pull/7693
> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>
> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин :
> >
> > Hello Nikolay,
> >
> > > Who deprecated visor and when? Maybe I miss something?
> > On the one hand, there was technically no community consensus that this
> > tool should be obsolete.
> > On the other hand, my opinion based on the following topic:
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> > Moreover, it seems to me, currently, the control utility is widely used
> and
> > actively developed, instead of the visor.
> >
> > > It's true that, for now, Ignite doesn't have "tool strategy" I think
> it's
> > a big issue from the user's point of view.
> > I absolutely agree with that.
> >
> > > We should solve it in the nearest time. Feel free to start this
> activity
> > I have no plan at the moment. However, at the first stage, we could
> > understand the difference between visor and control utility.
> > All useful features from visor should be moved/implemented in control
> > utility and after that visor tool and should be marked as
> > deprecated/obsoleted.
> >
> > > You need to throw in control.sh also, which does some kind of
> statistics
> > too, such as idle_verify.
> > > Please, clarify your idea:
> > >We should use some info from control.sh to the report?
> > >The report should be generated by some control.sh subcommand?
> > If I am not mistaken, the oracle database has AWR tool (mentioned on the
> > IEP page) which is a command-line utility that generates HTML reports.
> > I like this idea and I think this is a good approach that can be realized
> > in the control utility.
> > If we have a case that cannot be implemented in this way, we have to
> > clearly states the difference between these tools so as not to confuse
> our
> > users.
> > What do you think?
> >
> > Thanks,
> > Slava.
> >
> >
> > сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov :
> >
> > > Hello, Slava, Ilya, Denis.
> > >
> > > Thanks for joining this discussion!
> > >
> > > > - visor (which is deprecated)
> > >
> > > Who deprecated visor and when?
> > > Maybe I miss something?
> > >
> > > > - web-console (to be honest, I don't quite understand the status of
> this
> > > tool)
> > >
> > > +1.
> > >
> > > > I am not against the new tool, I just want to understand the
> motivation
> > > to not improve the existing sub-projects.
> > >
> > > It's true that, for now, Ignite doesn't have "tool strategy"
> > > I think it's a big issue from the user's point of view.
> > > We should solve it in the nearest time.
> > > Feel free to start this activity.
> > >
> > > > - new ignite-profiling (which is a monitoring tool as well, judging
> by
> > > the provided link [1] )
> > >
> > > The general idea is the following:
> > >
> > > 1. We should have some profiling mechanism that will generate a
> node-local
> > > event log
> > > 2. We should have a tool that can export events to some third-party
> > > system. This system can be an Elastic Search(Kibana) or Ignite
> performance
> > > report or Kafka log, whatever.
> > > 3. Ignite performance report, in the first release, should be a
> "static"
> > > tool.
> > > This means we take static logs(that is not rewritten in the
> analysis
> > > time) and feed them in the script(or tool or control.sh, whatever)
> > > The script produces static report that can be used for overall
> > > performance analysis.
> > >
> > > The primary users of this 

Re: Question: network issues of single node.

[Alexey Goncharuk]

Vladimir,

Such behavior can be introduced by an erroneous firewall configuration (I
can't find a link, but I remember that quite a large number of major
incidents are caused by an incorrect configuration change). If such a case
can be detected, we prefer Ignite to shutdown some of the nodes rather than
leave the whole cluster hanging on connection await.

сб, 6 июн. 2020 г. в 17:09, Denis Magda :

> Finally, I got your question.
>
> Back in 2017-2018, there was a Discovery SPI's stabilization activity. The
> networking component could fail in various hard-to-reproduce scenarios
> affecting cluster availability and consistency. That ticket reminds me of
> those notorious issues that would fire once a week or month under specific
> configuration settings. So, I would not touch the code that fixes the issue
> unless @Alexey Goncharuk  or @Sergey Chugunov
>  confirms that it's safe to do. Also, there should
> be a test for this scenario.
>
> -
> Denis
>
>
> On Fri, Jun 5, 2020 at 12:28 AM Vladimir Steshin 
> wrote:
>
> > Denis,
> >
> > I have no nodes that I'm unable to interconnect. This case is simulated
> > in IgniteDiscoveryMassiveNodeFailTest.testMassiveFailSelfKill()
> > Introduced in [1].
> >
> > I’m asking if it is real or supposed problem. Where it was met? Which
> > network configuration/issues could be?
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7163
> >
> > 05.06.2020 1:01, Denis Magda пишет:
> > > Vladimir,
> > >
> > > I'm suggesting to share the log files from the nodes that are unable to
> > > interconnect so that the community can check them for potential issues.
> > > Instead of sharing the logs from all the 5 nodes, try to start a
> > two-nodes
> > > cluster with the nodes that fail to discover each other and attach the
> > logs
> > > from those.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Jun 4, 2020 at 1:57 PM Vladimir Steshin 
> > wrote:
> > >
> > >> Denis, hi.
> > >>
> > >>   Sorry, I didn’t catch your idea. Are you saying this can happen
> > and
> > >> suggest experiment? I’m not descripting a probable case. It is already
> > >> done in [1]. I’m asking is it real, where it was met.
> > >>
> > >>
> > >> 04.06.2020 23:33, Denis Magda пишет:
> > >>> Vladimir,
> > >>>
> > >>> Please do the following experiment. Start a 2-nodes cluster booting
> > node
> > >> 3
> > >>> and, for instance, node 5. Those won't be able to interconnect
> > according
> > >> to
> > >>> your description. Attach the log files from both nodes for analysis.
> > This
> > >>> should be a networking issue.
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Thu, Jun 4, 2020 at 1:24 PM Vladimir Steshin 
> > >> wrote:
> > Hi, Igniters.
> > 
> > 
> > I wanted to ask how one node may not be able to connect to
> > another
> >  whereas rest of the cluster can. This got covered in [1]. In short:
> > node
> >  3 can't connect to nodes 4 and 5 but can to 1. At the same time,
> node
> > 2
> >  can connect to 4. Questions:
> > 
> >  1) Is it real case? Where this problem came from?
> > 
> >  2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t
> > connect
> >  to 4 (and 5) too?
> > 
> >  Sergey, Dmitry maybe you bring light (I see you in [1])? I'm
> >  participating in [2] and found this backward connection checking.
> >  Answering would help us a lot.
> > 
> >  Thanks!
> > 
> >  [1]
> >  https://issues.apache.org/jira/browse/IGNITE-7163<
> >  https://issues.apache.org/jira/browse/IGNITE-7163>
> > 
> >  [2]
> > 
> > 
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >  <
> > 
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >
>

Roadmap of new distributed SQL engine based on Calcite

[Юрий]

Dear Igniters,

Many of you heard about starting development of new SQL engine based on
Calcite. Currently we are only in beginning of the way and it will require
many works to achieve the goal. In order to understand where are we and
which features already done and which of them could be started by you will
be better to have some consolidate resource. For the reason I've prepare
roadmap page [1] , where we could track progress of development new SQL
engine.

Let's start fill the page and providing rough estimates. Dear, Ignite user
community, please share your suggestion as well.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Calcite-powered+SQL+Engine+Roadmap

-- 
Живи с улыбкой! :D

Re: IGNITE-12702 - Discussion

[Ilya Kasnacheev]

Hello!

Unfortunately your merged change is not visible in this PR (0 files
changed). Please create your pull request off your feature branch, not
master.

Regards,
-- 
Ilya Kasnacheev


вс, 7 июн. 2020 г. в 20:20, kartik somani :

> Hello,
>
> I have created a pull request for this issue
> <
> https://issues.apache.org/jira/browse/IGNITE-12702?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN
> >
> : https://github.com/apache/ignite/pull/7882
> Can someone review?
>
> Regards,
> Kartik
>
> On Mon, Apr 20, 2020 at 11:48 PM kartik somani <
> kartiksomani.offic...@gmail.com> wrote:
>
> > Hello Igniters,
> >
> > I have created pull request for IGNITE-12702. Can someone review this?
> > https://github.com/apache/ignite/pull/7663
> >
> > regards,
> > Kartik
> >
> > On Mon, Mar 23, 2020 at 11:28 PM Denis Magda  wrote:
> >
> >> Hi Kartik,
> >>
> >> Actually, it means quite the opposite. You should fill in the release
> >> notes
> >> field before the ticket is resolved/closed so that a release manager of
> a
> >> next Ignite version adds a proper line to the final release notes with
> all
> >> the changes.
> >>
> >> As for the docs, it's highly likely you don't need to update Ignite
> >> technical documentation (apacheignite.readme.io), thus, the field
> should
> >> be
> >> disabled. However, see if APIs (Javadocs) require an update. Hope this
> >> helps. Btw, this page elaborates more on our current docs process that
> is
> >> being revisited and improved soon:
> >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Sun, Mar 22, 2020 at 6:33 AM kartik somani <
> >> kartiksomani.offic...@gmail.com> wrote:
> >>
> >> > Hello Igniters,
> >> >
> >> > I want to start working on ticket IGNITE-12702
> >> > .  The flags on
> >> this
> >> > issue are "Docs Required, Release Notes Required". Does this mean I
> >> have to
> >> > first write/get docs and release notes before starting work on this?
> >> >
> >> > Regards,
> >> > Kartik
> >> >
> >>
> >
>

Re: [CVE-2020-1963] Apache Ignite access to file system disclosure vulnerability

[Юрий]

Denis,

It has been done in the same day as it announced here as described at
https://www.apache.org/security/committers.html#vulnerability-handling.
Probably it require some time to information to be updated.


Also I can confirm that no any plans to provide patch for any previous
versions of Ignite.



пт, 5 июн. 2020 г. в 19:20, Denis Magda :

> Yury,
>
> Could you please update the CVE with the details from this announcement?
>
> Nick, to my knowledge, there are no any plans to propagate this fix to the
> downstream versions such as 2.7, etc.
>
> -
> Denis
>
>
> On Wed, Jun 3, 2020 at 8:10 AM Nick Popov  wrote:
>
>> Are you going to provide CVE-2020-1964 patches and patch instructions for
>> previous Ignite versions?
>>
>>
>>
>> Regards,
>>
>> -Nick
>>
>>
>>
>> *From:* Sriveena Mattaparthi 
>> *Sent:* Wednesday, June 3, 2020 9:04 AM
>> *To:* u...@ignite.apache.org; dev ;
>> annou...@apache.org; Apache Security Team 
>> *Subject:* COMMERCIAL:RE: [CVE-2020-1963] Apache Ignite access to file
>> system disclosure vulnerability
>>
>>
>>
>> Thanks, Could you please confirm when the analysis will be updated here
>> for the CVE logged.
>>
>> https://nvd.nist.gov/vuln/detail/CVE-2020-1963
>>
>>
>>
>> Regards,
>> Sriveena
>>
>>
>>
>> *From:* Юрий 
>> *Sent:* 03 June 2020 16:02
>> *To:* dev ; u...@ignite.apache.org;
>> annou...@apache.org; Apache Security Team ;
>> Sriveena Mattaparthi 
>> *Subject:* [CVE-2020-1963] Apache Ignite access to file system
>> disclosure vulnerability
>>
>>
>>
>> Hi All,
>>
>> Apache Ignite 2.8.1 has been released. The release contain fix of
>> critical vulnerability
>>
>> CVE-2020-1963: Apache Ignite access to file system through predefined H2
>> SQL functions
>>
>> Severity: Critical
>>
>> Vendor:
>> The Apache Software Foundation
>>
>> Versions Affected:
>> All versions of Apache Ignite up to 2.8
>>
>> Impact
>> An attacker can use embedded H2 SQL functions to access a filesystem for
>> write and read.
>>
>> Description:
>> Apache Ignite uses H2 database to build SQL distributed execution engine.
>> H2 provides SQL functions which could be used by attacker to access to a
>> filesystem.
>>
>> Mitigation:
>> Ignite 2.8 or earlier users should upgrade to 2.8.1
>> In case SQL is not used at all the issue could be mitigated by removing
>> ignite-indexing.jar from Ignite classpath
>> Risk could be partially mitigated by using non privileged user to start
>> Apache Ignite.
>>
>> Credit:
>> This issue was discovered by Sriveena Mattaparthi of ekaplus.com
>> 
>>
>>
>>
>> --
>>
>> Живи с улыбкой! :D
>>
>> “Confidentiality Notice: The contents of this email message and any
>> attachments are intended solely for the addressee(s) and may contain
>> confidential and/or privileged information and may be legally protected
>> from disclosure. If you are not the intended recipient of this message or
>> their agent, or if this message has been addressed to you in error, please
>> immediately alert the sender by reply email and then delete this message
>> and any attachments. If you are not the intended recipient, you are hereby
>> notified that any use, dissemination, copying, or storage of this message
>> or its attachments is strictly prohibited.”
>>
>>
>>
>>  CAUTION EXTERNAL EMAIL - The email originated outside the organization.  Do 
>> not click on any links or open attachments unless you recognize the sender 
>> and know the content is safe.
>>
>>
>>
>>
>>
>>
>> TDECU and our subsidiaries are committed to maintaining Member 
>> confidentiality. Please note this message is being sent using a secure 
>> connection to ensure all information remains private and confidential. The 
>> information contained in this message is intended only for the recipient. If 
>> the reader of this message is not the intended recipient, please delete 
>> immediately.
>>
>>
>>
>>

-- 
Живи с улыбкой! :D

[jira] [Created] (IGNITE-13131) OpenCensusMetricExporterSpi and prometheus on server node

[Alexander Emelianov (Jira)]

Alexander Emelianov created IGNITE-13131:


 Summary: OpenCensusMetricExporterSpi and prometheus on server node
 Key: IGNITE-13131
 URL: https://issues.apache.org/jira/browse/IGNITE-13131
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.8.1, 2.8
Reporter: Alexander Emelianov


OpenCensusMetricExporterSpi didn't start prometheus exporter, which leads to 
the impossibility of configuration on the server node



--
This message was sent by Atlassian Jira
(v8.3.4#803005)