Re: [2.9.1]in SYS.METRICS View some data is always 0

[38797715]

Hello Ilya,

I got it.
If the table is created through JDBC, the metric data will be 0.
Only when the CREATE TABLE statement is executed through SqlFieldsQuery, 
the data will be counted.


I think it's a bug,Ilya, can you confirm again?

在 2021/1/30 上午12:06, Ilya Kasnacheev 写道:

Hello!

I can see some values on the server when executing SqlFieldsQuery on 
the same server:
| cache.foo.QueryCompleted   | 3  | 
   |
| cache.foo.QueryExecuted    | 3  | 
   |
| cache.foo.QueryFailed  | 0  | 
   |
| cache.foo.QueryMaximumTime | 350    | 
   |
| cache.foo.QueryMinimalTime | 101    | 
   |
| cache.foo.QuerySumTime | 665    | 
   |


I can also see them on client, but you need to a) specify 
ClientConnectorConfiguration on client node with non-default port, b) 
connect to that port with JDBC, and c) Enable metrics exporter SPI on 
the client node. Then I can see the same, after running 
SqlFieldQuery's on client:
| cache.foo.QueryCompleted   | 3  | 
   |
| cache.foo.QueryExecuted    | 3  | 
   |
| cache.foo.QueryFailed  | 0  | 
   |
| cache.foo.QueryMaximumTime | 269    | 
   |
| cache.foo.QueryMinimalTime | 20 | 
   |
| cache.foo.QuerySumTime | 424    | 
   |


Regards,
--
Ilya Kasnacheev


чт, 28 янв. 2021 г. в 14:42, 38797715 <38797...@qq.com 
>:


Hello Ilya,

if use sqlline execute sql,the feedback result of the following
statement is correct.

SELECT * FROM sys.metrics WHERE name LIKE 'sql%' ORDER BY name;

I used SqlFieldsQuery again to do the above tests, and the results
were all 0. I think there may be a issue in this.

在 2021/1/28 下午6:52, Ilya Kasnacheev 写道:

Hello!

I think these metrics will be gathered for
ScanQuery/SqlFieldsQuery executed via native Java Query API, but
they will not be gathered for statements executed via JDBC.

One obvious reason is that Java Query API's queries are bound to
a specific cache. JDBC query is not bound to specific cache: JDBC
query may operate on zero or more caches. We could map these
queries back to participating caches, but I don't see that we do
that.

You could still  use sql.queries.user. metrics:
SELECT * FROM sys.metrics WHERE name LIKE 'sql%' ORDER BY name;

Regards,
-- 
Ilya Kasnacheev



ср, 27 янв. 2021 г. в 15:29, 38797715 <38797...@qq.com
>:

Hello Ilya,

The test method is as follows:

Start 2 nodes on the localhost.
Use the CREATE TABLE statement to create a table;
Use the COPY command to load some data;
Access to cluster through sqlline;
Execute select count (*) from T;
Execute select * from sys.metrics  WHERE name LIKE '%cache.T%';
At this time, you will find that the relevant data are all 0,
but the value of OffHeapEntriesCount is still correct.

If you use sqlline to access another node, the result is the
same.

The configuration file to start the cluster is as follows:













































在 2021/1/27 下午6:24, Ilya Kasnacheev 写道:

Hello!

These values are per-node, as far as I know. Is it possible
that you have connected to a node which does not handle any
queries (as reducer, anyway)?

Regards,
-- 
Ilya Kasnacheev



вт, 26 янв. 2021 г. в 13:48, 38797715 <38797...@qq.com
>:

Hi,

We found that SYS.METRICS View some data is always 0,
such as

QueryCompleted,QueryExecuted,QuerySumTime,QueryCompleted,QuerySumTime

and QueryMaximumTime. This is a bug? Or what
configuration is needed? Or
the related functions have not been implemented yet?

Query System Views Ignite 2.9.1

[Nathan Neely]

How can I query system views from a server node? I have looked at example code 
for clients: 
https://ignite.apache.org/docs/latest/monitoring-metrics/system-views, but I 
can’t seem to find any examples for server nodes.
Can you use SqlFieldsQuery to return data from system views?

IgniteCache cache = getIgnite().cache("SYS");
if(cache != null){
SqlFieldsQuery sqlq = new SqlFieldsQuery("SELECT * from 
NODES").setSchema("SYS");
cache.query(sqlq).getAll();
}


Thanks,

Nathan Neely
Engineer IV, Software 5x Core
Benefitfocus for life™

Benefitfocus® (NASDAQ: BNFT) unifies the entire benefits industry on a single 
technology platform to protect consumers for life. Our powerful cloud-based 
software, data-driven insights and thoughtfully-designed services enable 
employers, insurance brokers, health plans and suppliers to simplify the 
complexity of benefits administration and deliver health, wealth, property and 
lifestyle products through a world-class benefits experience. Our mission is 
simple: to improve lives with benefits. Learn more at 
www.benefitfocus.com, 
LinkedIn and 
Twitter.

Re: rebalancing and jobs

[akorensh]

Hi,
  You can monitor rebalance programmatically using JMX,
  
https://ignite.apache.org/docs/latest/monitoring-metrics/metrics#monitoring-rebalancing
   or events:
https://ignite.apache.org/docs/latest/events/events#cache-rebalancing-events
   and have your app react accordingly.


   You can set the rebalanceDelay property to have each rebalance job start
w/a delay.
   https://ignite.apache.org/docs/latest/data-rebalancing#other-properties

   A rebalance is usually triggered in response to discovery or cache
lifecycle events:
   https://ignite.apache.org/docs/latest/events/events#discovery-events
  
https://ignite.apache.org/docs/latest/events/events#cache-lifecycle-events

   Note that w/persistent clusters, rebalance is triggered when there is a
change in the
   baseline topology: https://ignite.apache.org/docs/latest/data-rebalancing

   There is not a way to delay an upcoming rebalancing job save for the
aforementioned rebalanceDelay property. The property will delay all
rebalance jobs a set number of milliseconds.

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.9.1]in SYS.METRICS View some data is always 0

[Ilya Kasnacheev]

Hello!

I can see some values on the server when executing SqlFieldsQuery on the
same server:
| cache.foo.QueryCompleted   | 3  |
   |
| cache.foo.QueryExecuted| 3  |
   |
| cache.foo.QueryFailed  | 0  |
   |
| cache.foo.QueryMaximumTime | 350|
   |
| cache.foo.QueryMinimalTime | 101|
   |
| cache.foo.QuerySumTime | 665|
   |

I can also see them on client, but you need to a) specify
ClientConnectorConfiguration on client node with non-default port, b)
connect to that port with JDBC, and c) Enable metrics exporter SPI on the
client node. Then I can see the same, after running SqlFieldQuery's on
client:
| cache.foo.QueryCompleted   | 3  |
   |
| cache.foo.QueryExecuted| 3  |
   |
| cache.foo.QueryFailed  | 0  |
   |
| cache.foo.QueryMaximumTime | 269|
   |
| cache.foo.QueryMinimalTime | 20 |
   |
| cache.foo.QuerySumTime | 424|
   |

Regards,
-- 
Ilya Kasnacheev


чт, 28 янв. 2021 г. в 14:42, 38797715 <38797...@qq.com>:

> Hello Ilya,
>
> if use sqlline execute sql, the feedback result of the following
> statement is correct.
>
> SELECT * FROM sys.metrics WHERE name LIKE 'sql%' ORDER BY name;
>
> I used SqlFieldsQuery again to do the above tests, and the results were
> all 0. I think there may be a issue in this.
> 在 2021/1/28 下午6:52, Ilya Kasnacheev 写道:
>
> Hello!
>
> I think these metrics will be gathered for ScanQuery/SqlFieldsQuery
> executed via native Java Query API, but they will not be gathered for
> statements executed via JDBC.
>
> One obvious reason is that Java Query API's queries are bound to a
> specific cache. JDBC query is not bound to specific cache: JDBC query may
> operate on zero or more caches. We could map these queries back to
> participating caches, but I don't see that we do that.
>
> You could still  use sql.queries.user. metrics:
> SELECT * FROM sys.metrics WHERE name LIKE 'sql%' ORDER BY name;
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 27 янв. 2021 г. в 15:29, 38797715 <38797...@qq.com>:
>
>> Hello Ilya,
>>
>> The test method is as follows:
>>
>> Start 2 nodes on the localhost.
>> Use the CREATE TABLE statement to create a table;
>> Use the COPY command to load some data;
>> Access to cluster through sqlline;
>> Execute select count (*) from T;
>> Execute select * from sys.metrics  WHERE name LIKE '%cache.T%';
>> At this time, you will find that the relevant data are all 0, but the
>> value of OffHeapEntriesCount is still correct.
>>
>> If you use sqlline to access another node, the result is the same.
>>
>> The configuration file to start the cluster is as follows:
>> > "org.apache.ignite.configuration.IgniteConfiguration">
>> 
>> > "#{systemEnvironment['CONSISTENTID']}"/>
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> > "org.apache.ignite.configuration.CacheConfiguration">
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> > "org.apache.ignite.configuration.CacheConfiguration">
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 在 2021/1/27 下午6:24, Ilya Kasnacheev 写道:
>>
>> Hello!
>>
>> These values are per-node, as far as I know. Is it possible that you have
>> connected to a node which does not handle any queries (as reducer, anyway)?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> вт, 26 янв. 2021 г. в 13:48, 38797715 <38797...@qq.com>:
>>
>>> Hi,
>>>
>>> We found that SYS.METRICS View some data is always 0, such as
>>> QueryCompleted,QueryExecuted,QuerySumTime,QueryCompleted,QuerySumTime
>>> and QueryMaximumTime. This is a bug? Or what configuration is needed? Or
>>> the related functions have not been implemented yet?
>>>
>>>

Re: Re[4]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Ok... thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.7.6] Unsupported protocol version: -76 with thin client.

[Ilya Kasnacheev]

Thank you.

https://issues.apache.org/jira/browse/IGNITE-13401 for the curious.

Regards,
-- 
Ilya Kasnacheev


пт, 29 янв. 2021 г. в 12:04, Alex Plehanov :

> Ilya, you can find reproducer in test
> IgniteBinaryTest.testBinaryTypeWithIdOfMarshallerHeader
> In this thread, we are talking about a workaround for older versions where
> this issue is not fixed.
>
> пт, 29 янв. 2021 г. в 11:49, Ilya Kasnacheev :
>
>> Hello!
>>
>> Do you have reproducer for that entity name issue? I would very much like
>> to check that. Why do you need to add 'x' to it?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 28 янв. 2021 г. в 16:26, Maximiliano Gazquez <
>> maximiliano@gmail.com>:
>>
>>> So we fixed it like this
>>>
>>> private static String fixEntityName(Function
>>> nameToTypeId, String name) {
>>> String result = name;
>>> while ((nameToTypeId.apply(result) & 0x00FF) == 103) {
>>> result = "x" + result;
>>> }
>>> return result;
>>> }
>>> Where nameToTypeId is BinaryBasicIdMapper#typeId basically and it works.
>>>
>>> Now, is there any way to calculate the ClientConfiguration size? To
>>> avoid the issue mentioned in
>>> https://issues.apache.org/jira/browse/IGNITE-13401
>>>
>>> A lso, we were
>>> using thick client but we decided to migrate to thin client because we need
>>> interoperability between version. Is there any documentation about this?
>>> Detailing compatibility between client/server versions.
>>>
>>> Thanks!
>>> On 28 Jan 2021 9:08 -0300, Pavel Tupitsyn , wrote:
>>>
>>> Ilya,
>>>
>>> Normally you can use any combination of thin client and server versions,
>>> the highest common protocol version is negotiated automatically
>>> during the handshake.
>>>
>>> There are some exceptions to this - not all thin clients support very
>>> old protocols,
>>> but with recent versions it should work.
>>>
>>> On Thu, Jan 28, 2021 at 1:58 PM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 I don't think you can use a more recent version of thin client with
 older server version.
 New thin client features usually require support from the server side
 as well.

 Regards,
 --
 Ilya Kasnacheev


 ср, 27 янв. 2021 г. в 21:45, jjimeno :

> Hello,
>
> Thanks for your answer.
> The server is already in version 2.9.1 and the c++ thin client is from
> master branch to get transactions support.
>
> José
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re[4]: Performance of Ignite as key-value datastore. C++ Thin Client.

[Zhenya Stanilovsky]

I suppose you can estimate time of loading 1.2 million keys with 2 ignite nodes 
right now.
Just load 1200 keys 1000 times, i believe assumed time would be similar.
And i hope you understand the difference between lsm\btree and 
embeded\distributed system.  
I can`t give you a fix date right now, may be next week will be clearer here.
 
thanks ! 
 
>First off, thanks to you and Zhenya for your support
>
>I'm afraid that's not easy, as key's data are unrelated.
>
>As we are in an evaluation phase, we would prefer to wait for the fix and
>perform single node benchmarking in the meanwhile... Instead of grouping
>keys, we might split the big transaction into several smaller ones, as
>Zhenya suggested, but that will not give us the real timings anyway.
>
>According to Zhenya's mail, the problem has been identified and it seems you
>can solve it. I know it's difficult, but.. could you give a rough fix date?
>
>
>
>--
>Sent from:  http://apache-ignite-users.70518.x6.nabble.com/ 
 
 
 
 

data rebalancing and partition map exchange with persistence

[Alan Ward]

I'm using Ignite 2.9.1, a 5 node cluster with persistence enabled,
partitioned caches with 1 backup.

I'm a bit confused about the difference between data rebalancing and
partition map exchange in this context.

1. Does data rebalancing occur when a node leaves or joins, or only when
you manually change the baseline topology (assuming automatic baseline
adjustment is disabled)? Again, this is on a cluster with persistence
enabled.

2. Sometimes I look at the partition counts of a cache across all the nodes
using Arrays.stream(ignite.affinity(cacheName).primaryPartitions(severNode)
and I see 0 partitions on one or even two nodes for some of the caches.
After a while it returns to a balanced state. What's going on here? Is this
data rebalancing at work, or is this the result of the partition map
exchange process determining that one node is/was down and thus switching
to use the backup partitions?

3. Is there a way to manually invoke the partition map exchange process? I
figured it would happen on cluster restart, but even after restarting the
cluster and seeing all baseline nodes connect I still observe the partition
imbalance. It often takes hours for this to resolve.

4. Sometimes I see 'partition lost' errors. If i am using persistence and
all the baseline nodes are online and connected, is it safe to assume no
data has been lost and just call cache.resetLostPartitions(myCaches)? Is
there a way calling that method would lead to data loss with persistence
enabled?

thanks for your help!

Re: Re[2]: Detecting checkpoints programmatically

[Maxim Muzafarov]

Hello Raymon,

Is it possible for your case writing a custom MetricExporterSpi [1]? So all
these checkpoint metrics [2] will be exported to your monitoring system.

It is also possible to get all metrics through the console [3].

[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/PushMetricsExporterAdapter.java
[2] https://issues.apache.org/jira/browse/IGNITE-13845
[3] https://issues.apache.org/jira/browse/IGNITE-13488

On Fri, 29 Jan 2021 at 09:15, Zhenya Stanilovsky  wrote:

> hi, check these links.
> [1] https://ignite.apache.org/docs/latest/monitoring-metrics/metrics
> [2]
> https://www.zylk.net/en/web-2-0/blog/-/blogs/kibana-dashboard-for-monitoring-liferay-jvm-via-jmx
>
>
>
> Hi Zhenya,
>
> It seems those events are not so useful as you suggest, [2]
> In the case of [1], how are these events accessible from a C# Ignite
> client?
>
> We don't currently use zabbix, but do use fluentd/kibana.
>
> Thanks,
> Raymond.
>
>
>
> On Fri, Jan 29, 2021 at 4:54 PM Zhenya Stanilovsky  > wrote:
>
>
> hi ! hope this would be helpful [1].
> Mentioned events is other one [2] = not useful for you.
> Also zabbix can parse the logs, you can obtain all cp info there.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13845
> [2]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/checkpoint/CheckpointSpi.html
>
>
>
> I want to make my system detect when a check point has occurred for
> monitoring and control purposes.
>
> I found the following events defined in the IA source:
>
> /**
>  * All checkpoint events. This array can be directly passed into
>  * {@link IgniteEvents#localListen(IgnitePredicate, int...)} method to
>  * subscribe to all checkpoint events.
>  *
>  * @see CheckpointEvent
>  */
> public static final int[] EVTS_CHECKPOINT = {
> EVT_CHECKPOINT_SAVED,
> EVT_CHECKPOINT_LOADED,
> EVT_CHECKPOINT_REMOVED
> };
>
> Are these the appropriate events to listen to?
>
> Thanks,
> Raymond.
>
> --
> 
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> raymond_wil...@trimble.com
> 
>
>
>
> 
>
>
>
>
>
>
>
>
> --
> 
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> raymond_wil...@trimble.com
> 
>
>
>
> 
>
>
>
>
>
>

Re: Ignite service

[Ilya Kasnacheev]

Hello!

You can run code on both server or client node, so it is up to you to
decide where do you want to call deploy.

No, you cannot register C# services using Spring XML

Regards,
-- 
Ilya Kasnacheev


вт, 19 янв. 2021 г. в 00:54, Jigna :

> Hi,
>
> I have another question for Ignite service:
>
> 1. If I will write services.deploy* method to deploy my service, how to
> write this code on the server node? Or I need to create a new client node
> and I need to register this service using "services.deploy*" code using my
> client node?
>
> 2. Just to confirm, as I can not register my service using ignite xml
> configuration, so I can not use below code right?
>
> 
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.snippets.services.MyCounterServiceImpl"/>
> 
> 
> 
> 
> 
>
> Thanks,
> Jigna
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: [2.7.6] Unsupported protocol version: -76 with thin client.

[Alex Plehanov]

Ilya, you can find reproducer in test
IgniteBinaryTest.testBinaryTypeWithIdOfMarshallerHeader
In this thread, we are talking about a workaround for older versions where
this issue is not fixed.

пт, 29 янв. 2021 г. в 11:49, Ilya Kasnacheev :

> Hello!
>
> Do you have reproducer for that entity name issue? I would very much like
> to check that. Why do you need to add 'x' to it?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 28 янв. 2021 г. в 16:26, Maximiliano Gazquez <
> maximiliano@gmail.com>:
>
>> So we fixed it like this
>>
>> private static String fixEntityName(Function
>> nameToTypeId, String name) {
>> String result = name;
>> while ((nameToTypeId.apply(result) & 0x00FF) == 103) {
>> result = "x" + result;
>> }
>> return result;
>> }
>> Where nameToTypeId is BinaryBasicIdMapper#typeId basically and it works.
>>
>> Now, is there any way to calculate the ClientConfiguration size? To avoid
>> the issue mentioned in https://issues.apache.org/jira/browse/IGNITE-13401
>>
>> A lso, we were using
>> thick client but we decided to migrate to thin client because we need
>> interoperability between version. Is there any documentation about this?
>> Detailing compatibility between client/server versions.
>>
>> Thanks!
>> On 28 Jan 2021 9:08 -0300, Pavel Tupitsyn , wrote:
>>
>> Ilya,
>>
>> Normally you can use any combination of thin client and server versions,
>> the highest common protocol version is negotiated automatically
>> during the handshake.
>>
>> There are some exceptions to this - not all thin clients support very old
>> protocols,
>> but with recent versions it should work.
>>
>> On Thu, Jan 28, 2021 at 1:58 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I don't think you can use a more recent version of thin client with
>>> older server version.
>>> New thin client features usually require support from the server side as
>>> well.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 27 янв. 2021 г. в 21:45, jjimeno :
>>>
 Hello,

 Thanks for your answer.
 The server is already in version 2.9.1 and the c++ thin client is from
 master branch to get transactions support.

 José



 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

>>>

Re: [2.7.6] Unsupported protocol version: -76 with thin client.

[Ilya Kasnacheev]

Hello!

Do you have reproducer for that entity name issue? I would very much like
to check that. Why do you need to add 'x' to it?

Regards,
-- 
Ilya Kasnacheev


чт, 28 янв. 2021 г. в 16:26, Maximiliano Gazquez :

> So we fixed it like this
>
> private static String fixEntityName(Function
> nameToTypeId, String name) {
> String result = name;
> while ((nameToTypeId.apply(result) & 0x00FF) == 103) {
> result = "x" + result;
> }
> return result;
> }
> Where nameToTypeId is BinaryBasicIdMapper#typeId basically and it works.
>
> Now, is there any way to calculate the ClientConfiguration size? To avoid
> the issue mentioned in https://issues.apache.org/jira/browse/IGNITE-13401
>
> A lso, we were using
> thick client but we decided to migrate to thin client because we need
> interoperability between version. Is there any documentation about this?
> Detailing compatibility between client/server versions.
>
> Thanks!
> On 28 Jan 2021 9:08 -0300, Pavel Tupitsyn , wrote:
>
> Ilya,
>
> Normally you can use any combination of thin client and server versions,
> the highest common protocol version is negotiated automatically
> during the handshake.
>
> There are some exceptions to this - not all thin clients support very old
> protocols,
> but with recent versions it should work.
>
> On Thu, Jan 28, 2021 at 1:58 PM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> I don't think you can use a more recent version of thin client with older
>> server version.
>> New thin client features usually require support from the server side as
>> well.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 27 янв. 2021 г. в 21:45, jjimeno :
>>
>>> Hello,
>>>
>>> Thanks for your answer.
>>> The server is already in version 2.9.1 and the c++ thin client is from
>>> master branch to get transactions support.
>>>
>>> José
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>