Continuous Queries with several remote filter on the same cache

[Roman.Koriakov]

Hi Community,

I ask this question here because I haven't found the answer in the 
documentation.

Could you please clarify how Continuous Queries work? What the behavior of 
Continuous Queries if we have several clients with different Remote Filters on 
the same cache? For example, if we have: one server node with cache and we have 
up to 20 client nodes each of them will execute Continuous Query on the same 
cache but with different Remote Filters. Will each client get the data 
according to its remote filter? Or it is supposed to have only one Remote 
Filter for all clients and every client should filter data in its local event 
listener?
I would be grateful if you send some link which describes the behavior of 
Continuous Queries more thoroughly.
Best regards,
Roman

Re: Client transactions withLabel

[Guru Stron]

Thank you Alex !

On Fri, 5 Jun 2020 at 01:23, Alex Plehanov  wrote:

> Hello,
>
> The transaction label is helpful for diagnostic and exposed to some
> diagnostic tools (SYS.TRANSACTIONS system view, control.sh commands, JMX
> TransactionsMXBean, long-running transactions dump in logs), so you should
> be able to set it by thin client, to view it later by these tools.
>
> пт, 5 июн. 2020 г. в 00:49, Guru Stron :
>
> > Hi Igniters,
> >
> > I'm trying to implement thin client transactions for .NET and was
> wondering
> > what is the point of transaction label in thin clients? I've browsed a
> > little bit through Java implementation and don't see where it is used
> cause
> > it seems that it can be only set and there is no public APi to access it.
> >
> > Thank you!
> >
>

Re: Client transactions withLabel

[Alex Plehanov]

Hello,

The transaction label is helpful for diagnostic and exposed to some
diagnostic tools (SYS.TRANSACTIONS system view, control.sh commands, JMX
TransactionsMXBean, long-running transactions dump in logs), so you should
be able to set it by thin client, to view it later by these tools.

пт, 5 июн. 2020 г. в 00:49, Guru Stron :

> Hi Igniters,
>
> I'm trying to implement thin client transactions for .NET and was wondering
> what is the point of transaction label in thin clients? I've browsed a
> little bit through Java implementation and don't see where it is used cause
> it seems that it can be only set and there is no public APi to access it.
>
> Thank you!
>

Re: Question: network issues of single node.

[Denis Magda]

Vladimir,

I'm suggesting to share the log files from the nodes that are unable to
interconnect so that the community can check them for potential issues.
Instead of sharing the logs from all the 5 nodes, try to start a two-nodes
cluster with the nodes that fail to discover each other and attach the logs
from those.

-
Denis


On Thu, Jun 4, 2020 at 1:57 PM Vladimir Steshin  wrote:

> Denis, hi.
>
>  Sorry, I didn’t catch your idea. Are you saying this can happen and
> suggest experiment? I’m not descripting a probable case. It is already
> done in [1]. I’m asking is it real, where it was met.
>
>
> 04.06.2020 23:33, Denis Magda пишет:
> > Vladimir,
> >
> > Please do the following experiment. Start a 2-nodes cluster booting node
> 3
> > and, for instance, node 5. Those won't be able to interconnect according
> to
> > your description. Attach the log files from both nodes for analysis. This
> > should be a networking issue.
> >
> > -
> > Denis
> >
> >
> > On Thu, Jun 4, 2020 at 1:24 PM Vladimir Steshin 
> wrote:
> >
> >>   Hi, Igniters.
> >>
> >>
> >>   I wanted to ask how one node may not be able to connect to another
> >> whereas rest of the cluster can. This got covered in [1]. In short: node
> >> 3 can't connect to nodes 4 and 5 but can to 1. At the same time, node 2
> >> can connect to 4. Questions:
> >>
> >> 1) Is it real case? Where this problem came from?
> >>
> >> 2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t connect
> >> to 4 (and 5) too?
> >>
> >> Sergey, Dmitry maybe you bring light (I see you in [1])? I'm
> >> participating in [2] and found this backward connection checking.
> >> Answering would help us a lot.
> >>
> >> Thanks!
> >>
> >> [1]
> >> https://issues.apache.org/jira/browse/IGNITE-7163<
> >> https://issues.apache.org/jira/browse/IGNITE-7163>
> >>
> >> [2]
> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >> <
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >>
>

Client transactions withLabel

[Guru Stron]

Hi Igniters,

I'm trying to implement thin client transactions for .NET and was wondering
what is the point of transaction label in thin clients? I've browsed a
little bit through Java implementation and don't see where it is used cause
it seems that it can be only set and there is no public APi to access it.

Thank you!

Re: Question: network issues of single node.

[Vladimir Steshin]

Denis, hi.

    Sorry, I didn’t catch your idea. Are you saying this can happen and 
suggest experiment? I’m not descripting a probable case. It is already 
done in [1]. I’m asking is it real, where it was met.



04.06.2020 23:33, Denis Magda пишет:

Vladimir,

Please do the following experiment. Start a 2-nodes cluster booting node 3
and, for instance, node 5. Those won't be able to interconnect according to
your description. Attach the log files from both nodes for analysis. This
should be a networking issue.

-
Denis


On Thu, Jun 4, 2020 at 1:24 PM Vladimir Steshin  wrote:


  Hi, Igniters.


  I wanted to ask how one node may not be able to connect to another
whereas rest of the cluster can. This got covered in [1]. In short: node
3 can't connect to nodes 4 and 5 but can to 1. At the same time, node 2
can connect to 4. Questions:

1) Is it real case? Where this problem came from?

2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t connect
to 4 (and 5) too?

Sergey, Dmitry maybe you bring light (I see you in [1])? I'm
participating in [2] and found this backward connection checking.
Answering would help us a lot.

Thanks!

[1]
https://issues.apache.org/jira/browse/IGNITE-7163<
https://issues.apache.org/jira/browse/IGNITE-7163>

[2]

https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
<
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up

Re: Question: network issues of single node.

[Denis Magda]

Vladimir,

Please do the following experiment. Start a 2-nodes cluster booting node 3
and, for instance, node 5. Those won't be able to interconnect according to
your description. Attach the log files from both nodes for analysis. This
should be a networking issue.

-
Denis


On Thu, Jun 4, 2020 at 1:24 PM Vladimir Steshin  wrote:

>  Hi, Igniters.
>
>
>  I wanted to ask how one node may not be able to connect to another
> whereas rest of the cluster can. This got covered in [1]. In short: node
> 3 can't connect to nodes 4 and 5 but can to 1. At the same time, node 2
> can connect to 4. Questions:
>
> 1) Is it real case? Where this problem came from?
>
> 2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t connect
> to 4 (and 5) too?
>
> Sergey, Dmitry maybe you bring light (I see you in [1])? I'm
> participating in [2] and found this backward connection checking.
> Answering would help us a lot.
>
> Thanks!
>
> [1]
> https://issues.apache.org/jira/browse/IGNITE-7163<
> https://issues.apache.org/jira/browse/IGNITE-7163>
>
> [2]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >
>
>

Question: network issues of single node.

[Vladimir Steshin]

    Hi, Igniters.


    I wanted to ask how one node may not be able to connect to another 
whereas rest of the cluster can. This got covered in [1]. In short: node 
3 can't connect to nodes 4 and 5 but can to 1. At the same time, node 2 
can connect to 4. Questions:


1) Is it real case? Where this problem came from?

2) If node 3 can’t connect to 4 and 5, does it mean node 2 can’t connect 
to 4 (and 5) too?


Sergey, Dmitry maybe you bring light (I see you in [1])? I'm 
participating in [2] and found this backward connection checking. 
Answering would help us a lot.


Thanks!

[1] 
https://issues.apache.org/jira/browse/IGNITE-7163


[2] 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up

Re: Help with ignite cluster

[C Ravikiran]

I am able to see Topology snapshot in my logs as ver=23 servers=2
offheap=93 gb heap=120 gb

I am able see 2 severs.

Still we are facing the issue.

Could you please help me on this issue.

On Thu, 4 Jun, 2020, 5:34 pm Ilya Kasnacheev, 
wrote:

> Hello!
>
> I think that your nodes were not able to discover each other and formed two
> clusters.
>
> Can you find "Topology snapshot" messages from their logs, share those with
> us?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 4 июн. 2020 г. в 13:28, C Ravikiran :
>
> > Hi Team,
> >
> > Initially my application running with single node.
> >
> > Recently added one more node in to the cluster and my application is
> > working with two nodes without issues and also fetching data in cache IDs
> > also.
> >
> > But today one node was down which was runnig intially.
> >
> > Newly added node was running fine.
> >
> > But we are not able to see the data from newly added node.
> >
> > Two nodes sync has not done properly.
> >
> > We are using ignite 2.6 version
> >
> > Cache mode: partitioned, FULL_SYNC
> >
> > Could you please help me on this.
> > Ignite ui i am getting below error:
> > Fail to find data nodes for cache: *
> >
> > Thanks in advance
> > advanc. e
> >
>

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[ткаленко кирилл]

Folks

I suggest closing the ticket with a comment:

To kill process with exit code [1], you can use [2] by setting correct values 
of constructor arguments.

Objections?

1 - 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignition.html#KILL_EXIT_CODE
2 - org.apache.ignite.failure.StopNodeOrHaltFailureHandler

02.06.2020, 16:59, "ткаленко кирилл" :
> Hello, Alexey!
>
> I didn't quite understand about merge.
>
> If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we don't 
> stop node by timeout, we will terminate jvm.
>
> Or do you suggest only stopping the node in StopNodeFailureHandler and 
> terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?
>
> 02.06.2020, 16:46, "Alexey Goncharuk" :
>>>   > How exactly do you want to change the StopNodeFH?
>>>   I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
>>>   argument of JVM option) for disabling JVM termination.
>>
>>  When the flag is enabled, the behavior is identical to StopNodeOrHaltFH
>>  with tryStop=false. In other words,
>>  StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
>>  tryStop=false
>>  StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
>>  tryStop=true
>>
>>  Why have two different failure handlers with identical behavior? Perhaps,
>>  there is confusion and we should consider merging these classes into one?

Re: Help with ignite cluster

[C Ravikiran]

Thank you for the response.
I have one cluster and two nodes.  Both nodes I am able to see ignite UI.

Still I am facing the same problem.
Could please let me know, any configuration I have missed.


On Thu, 4 Jun, 2020, 5:34 pm Ilya Kasnacheev, 
wrote:

> Hello!
>
> I think that your nodes were not able to discover each other and formed two
> clusters.
>
> Can you find "Topology snapshot" messages from their logs, share those with
> us?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 4 июн. 2020 г. в 13:28, C Ravikiran :
>
> > Hi Team,
> >
> > Initially my application running with single node.
> >
> > Recently added one more node in to the cluster and my application is
> > working with two nodes without issues and also fetching data in cache IDs
> > also.
> >
> > But today one node was down which was runnig intially.
> >
> > Newly added node was running fine.
> >
> > But we are not able to see the data from newly added node.
> >
> > Two nodes sync has not done properly.
> >
> > We are using ignite 2.6 version
> >
> > Cache mode: partitioned, FULL_SYNC
> >
> > Could you please help me on this.
> > Ignite ui i am getting below error:
> > Fail to find data nodes for cache: *
> >
> > Thanks in advance
> > advanc. e
> >
>

Re: Help with ignite cluster

[Ilya Kasnacheev]

Hello!

I think that your nodes were not able to discover each other and formed two
clusters.

Can you find "Topology snapshot" messages from their logs, share those with
us?

Regards,
-- 
Ilya Kasnacheev


чт, 4 июн. 2020 г. в 13:28, C Ravikiran :

> Hi Team,
>
> Initially my application running with single node.
>
> Recently added one more node in to the cluster and my application is
> working with two nodes without issues and also fetching data in cache IDs
> also.
>
> But today one node was down which was runnig intially.
>
> Newly added node was running fine.
>
> But we are not able to see the data from newly added node.
>
> Two nodes sync has not done properly.
>
> We are using ignite 2.6 version
>
> Cache mode: partitioned, FULL_SYNC
>
> Could you please help me on this.
> Ignite ui i am getting below error:
> Fail to find data nodes for cache: *
>
> Thanks in advance
> advanc. e
>

[jira] [Created] (IGNITE-13119) Data race when getting index rebuild status

[Kirill Tkalenko (Jira)]

Kirill Tkalenko created IGNITE-13119:


 Summary: Data race when getting index rebuild status
 Key: IGNITE-13119
 URL: https://issues.apache.org/jira/browse/IGNITE-13119
 Project: Ignite
  Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.9






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Help with ignite cluster

[C Ravikiran]

Hi Team,

Initially my application running with single node.

Recently added one more node in to the cluster and my application is
working with two nodes without issues and also fetching data in cache IDs
also.

But today one node was down which was runnig intially.

Newly added node was running fine.

But we are not able to see the data from newly added node.

Two nodes sync has not done properly.

We are using ignite 2.6 version

Cache mode: partitioned, FULL_SYNC

Could you please help me on this.
Ignite ui i am getting below error:
Fail to find data nodes for cache: *

Thanks in advance
advanc. e

Re: [DISCUSS] Make AtomicConfiguration more flexible

[Vladimir Pligin]

I've raised the issue https://issues.apache.org/jira/browse/IGNITE-13118.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13118) Make AtomicConfiguration more customizable

[Vladimir Pligin (Jira)]

Vladimir Pligin created IGNITE-13118:


 Summary: Make AtomicConfiguration more customizable
 Key: IGNITE-13118
 URL: https://issues.apache.org/jira/browse/IGNITE-13118
 Project: Ignite
  Issue Type: Improvement
  Components: data structures
Affects Versions: 2.8.1
Reporter: Vladimir Pligin
Assignee: Vladimir Pligin


*Description of the issue:*

_AtomicConfiguration_ 
([https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/AtomicConfiguration.html])
 is not flexible enough to cover some reasonable scenarios at the moment. It's 
possible to configure cache mode, group name, affinity and backups of the 
underlying cache only. For example there's no straightforward way to configure 
a node filter. It can be handful as sometimes it's required to avoid atomic 
sequence residence on a subset of nodes.

*Proposed solution:*

Make injection of a full-fledged _CacheConfiguration_ possible during a 
construction process of _AtomicConfiguration_. . 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Extra test coverage for ACTIVE_READ_ONLY cluster state

[Sergey Antonov]

Igniters, I faced several problems during write tests for the read-only
mode:

   1. You can create/destroy cache on the read-only cluster. Fixed in [1].
   2. The read-only mode doesn't affect LOCAL caches. Ticket created [2].
   3. IgniteCache#isClosed() doesn't work on server nodes. Ticket created
   [3].

Also, I wrote tests for:

   1. IgniteCache API (including create(), destroy(), invoke())
   2. Near caches.
   3. 3rd party CacheStore.
   4. Services (deploy, execution, cancel)
   5. Datastructures (IgniteAtomicLong, IgniteAtomicReference,
   IgniteAtomicSequence, IgniteAtomicStamped, IgniteCountDownLatch,
   IgniteQueue, IgniteSet)
   6. Updates to meta storage.

The patch [4] ready for review.

[1] https://issues.apache.org/jira/browse/IGNITE-13071
[2] https://issues.apache.org/jira/browse/IGNITE-13076
[3] https://issues.apache.org/jira/browse/IGNITE-13102
[4] https://github.com/apache/ignite/pull/7853/files


вт, 26 мая 2020 г. в 20:28, Sergey Antonov :

> Maxim, I've created a ticket [1] for this change.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13079
>
> вт, 26 мая 2020 г. в 20:09, Sergey Antonov :
>
>> Maxim, I'd prefer to do this with a separate ticket.
>>
>> вт, 26 мая 2020 г. в 19:59, Maxim Muzafarov :
>>
>>> Sergey,
>>>
>>> Sounds good!
>>> Should we consider removing the deprecated methods `active()`,
>>> `active(boolean active)` from tests also?
>>>
>>> On Tue, 26 May 2020 at 12:18, Sergey Antonov 
>>> wrote:
>>> >
>>> > Hello, Igniters.
>>> >
>>> > I introduced cluster read-only mode [1] and a new API for cluster state
>>> > change [2]. At the moment we don't have good test coverage for this
>>> > feature. I'm going to fix it and write tests and check that operations
>>> > are *denied
>>> > *in read-only mode:
>>> >
>>> >- data structures usage
>>> >- cache create/clear/destroy
>>> >- DDL requests
>>> >- cache updates by task's execution / deployed service
>>> >
>>> > And the following operations are *allowed *in read-only mode:
>>> >
>>> >- update of metastorage / distributed metastorage
>>> >- updates to ignite-sys-cache
>>> >- task's execution, but updates must be rejected
>>> >- service deploy/undeploy, but updates must be rejected
>>> >- data recovery on node join
>>> >
>>> > I'll work under these tests in ticket [3].
>>> > Any objections?
>>> >
>>> > [1] https://issues.apache.org/jira/browse/IGNITE-11256
>>> > [2] https://issues.apache.org/jira/browse/IGNITE-12225
>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13071
>>> > --
>>> > BR, Sergey Antonov
>>>
>>
>>
>> --
>> BR, Sergey Antonov
>>
>
>
> --
> BR, Sergey Antonov
>


-- 
BR, Sergey Antonov

[jira] [Created] (IGNITE-13117) Calcite integration. Update Calcite version to 1.23.0

[Igor Seliverstov (Jira)]

Igor Seliverstov created IGNITE-13117:
-

 Summary: Calcite integration. Update Calcite version to 1.23.0
 Key: IGNITE-13117
 URL: https://issues.apache.org/jira/browse/IGNITE-13117
 Project: Ignite
  Issue Type: Improvement
Reporter: Igor Seliverstov
Assignee: Igor Seliverstov


There are two important features in the new Calcite release what we need to 
solve current planner issues:  top-down traits propagation and bottom-up traits 
derivation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Tool for performance statistics reports

[Nikita Amelchev]

Hi, Igniters.

I have implemented cluster profiling and tool to build the performance
report. It's ready to be reviewed. [1, 2]

Profiling can be managed by JMX bean. I have plans to implement it to
control.sh also.

Nodes write statistics to the temporary off heap buffer and then one
thread flushes to the profiling files. The write mechanics and format
is like WAL logging.
The report contains the following statistics:
- nodes and caches info
- cache operations and transaction statistics
- SQL and scan queries statistics (include logical and physical reads per query)
- tasks and jobs statistics.

More details in the IEP [3].

[1] https://github.com/apache/ignite/pull/7693
[2] https://issues.apache.org/jira/browse/IGNITE-12666
[3] 
https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool

вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин :
>
> Hello Nikolay,
>
> > Who deprecated visor and when? Maybe I miss something?
> On the one hand, there was technically no community consensus that this
> tool should be obsolete.
> On the other hand, my opinion based on the following topic:
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> Moreover, it seems to me, currently, the control utility is widely used and
> actively developed, instead of the visor.
>
> > It's true that, for now, Ignite doesn't have "tool strategy" I think it's
> a big issue from the user's point of view.
> I absolutely agree with that.
>
> > We should solve it in the nearest time. Feel free to start this activity
> I have no plan at the moment. However, at the first stage, we could
> understand the difference between visor and control utility.
> All useful features from visor should be moved/implemented in control
> utility and after that visor tool and should be marked as
> deprecated/obsoleted.
>
> > You need to throw in control.sh also, which does some kind of statistics
> too, such as idle_verify.
> > Please, clarify your idea:
> >We should use some info from control.sh to the report?
> >The report should be generated by some control.sh subcommand?
> If I am not mistaken, the oracle database has AWR tool (mentioned on the
> IEP page) which is a command-line utility that generates HTML reports.
> I like this idea and I think this is a good approach that can be realized
> in the control utility.
> If we have a case that cannot be implemented in this way, we have to
> clearly states the difference between these tools so as not to confuse our
> users.
> What do you think?
>
> Thanks,
> Slava.
>
>
> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov :
>
> > Hello, Slava, Ilya, Denis.
> >
> > Thanks for joining this discussion!
> >
> > > - visor (which is deprecated)
> >
> > Who deprecated visor and when?
> > Maybe I miss something?
> >
> > > - web-console (to be honest, I don't quite understand the status of this
> > tool)
> >
> > +1.
> >
> > > I am not against the new tool, I just want to understand the motivation
> > to not improve the existing sub-projects.
> >
> > It's true that, for now, Ignite doesn't have "tool strategy"
> > I think it's a big issue from the user's point of view.
> > We should solve it in the nearest time.
> > Feel free to start this activity.
> >
> > > - new ignite-profiling (which is a monitoring tool as well, judging by
> > the provided link [1] )
> >
> > The general idea is the following:
> >
> > 1. We should have some profiling mechanism that will generate a node-local
> > event log
> > 2. We should have a tool that can export events to some third-party
> > system. This system can be an Elastic Search(Kibana) or Ignite performance
> > report or Kafka log, whatever.
> > 3. Ignite performance report, in the first release, should be a "static"
> > tool.
> > This means we take static logs(that is not rewritten in the analysis
> > time) and feed them in the script(or tool or control.sh, whatever)
> > The script produces static report that can be used for overall
> > performance analysis.
> >
> > The primary users of this report is a developer of Ignite based
> > applications and performance engineers.
> >
> > Ilya,
> >
> > > You need to throw in control.sh also, which does some kind of statistics
> > too, such as idle_verify.
> >
> > Please, clarify your idea:
> > We should use some info from control.sh to the report?
> > The report should be generated by some control.sh subcommand?
> >
> >
> > Denis,
> >
> > > Speaking of the probes/statistics collection approach, is it supposed to
> > reuse tracing capabilities that are to be added as part of IEP-35?
> >
> > For now, we don't have any results of tracing development available in
> > Apache Ignite.
> > Hopefully, we got some in a couple of weeks.
> > After it, we can start a discussion of how to merge two improvements.
> >
> >
> >
> > > 24 апр. 2020 г., в 20:32, Denis Magda  написал(а):
> > >
> > >>
> > >> Tracing is more deeply takes statistics. If it will be possible, I'm for
> > >>