Re: Issue with large number of Continuous Queries

[Denis Magda]

Hello,

How do you decide about a number of continuous queries to be started? I see
on one of your pictures that a query is started per key. Usually, a single
query per cache per client should be enough as long as the filter allows to
exclude records of no interest for a client. Probably, you can put
client-specific logic into the filter.

As for 1000s of continuous queries for a single node cluster, this sounds
like overkill as long as the server node needs to process all the 1000s
queries for every data change.

-
Denis


On Tue, Feb 11, 2020 at 3:00 AM zork  wrote:

> *Topology*:
>
> Server-1 --> Cache myCache, holds continuously updating data like market
> data(prices, status, tradetime etc) for instruments. InstrumentId is the
> key
> for this cache.
>Server-1 running with following jvm params:
> -Xms1g,-Xmx6g
>
> Client-1 --> Pushing continuous updates on the cache
>
> Client - 2 & 3 --> Listening updates on myCache using separate Continuous
> query on every key (i.e. one continuous query per instrumentId).
>
> The Cache Configuration is as follows:
>   cacheModePARTITIONED
>   atomicityModeATOMIC
>   backups2
>   readFromBackuptrue
>   copyOnReadtrue
>   statisticsEnabledtrue
>   managementEnabledtrue
>
> System hardware: 8 core, 32gb RAM
> For now all servers and client below run on same machine.
>
> -
>
> *Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
> updates per second on every item. Lets say both client-2 and client-3 have
> 1000 different continuous queries open to listen to every update.
>
> With the above scenario, we are observing the server-1 alone taking 60-70%
> CPU and about 4GB memory.
> In this case when high number continuous queries machine reaches 100% CPU
> utilization.
>
> *Thinking to fix as*: Use single continuous query per client to listen to
> all the updates. i.e. there would be just one continuous query and it would
> listen to all the updates.
>
> 
>
> But now the problem is that both the clients do not need to listen to
> updates in all the keys in cache. So we are thinking of adding another
> column to the ignite cache using which we can filter the updates by
> checking
> if the client column of the updated row contains the client name for which
> filter is being checked. e.g. the new table would look like-
>
> 
>
> Would this be the correct way to achieve what we are trying to achieve? Or
> could this be done some other better way in ignite?
>
> Follow up Question:
> How many continuous queries can ignite handle at once with the
> configuration
> we mentioned or is there any such benchmark values available on any
> particular configuration? Is it fine to have as many as 1000 (or even more)
> continuous queries running at once? If yes, how can we make it less CPU
> intensive and more performant?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Issue with large number of Continuous Queries

[Andrei Aleksandrov]

Hi,

Please read some comments below:

1)You said:

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

Continues query can handle all updates for some cache. You shouldn't 
start special CQ for every entry. Please read the following:


https://apacheignite.readme.io/docs/continuous-queries

2)You said:

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when a high number of continuous queries machine reaches 
100% CPU

utilization.

You can try to monitor what Ignite parts take the CPU and memory usage. 
Taking into account the point 1) - you shouldn't have a lot of CQ for 
the same cache.


3)You said:

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

This is the correct way.

4)You said:

But now the problem is that both the clients do not need to listen to
updates in all the keys in the cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

Correct you can filter the updates using the following:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/AbstractContinuousQuery.html#getRemoteFilterFactory--

5)You said:

How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?

I don't think that the current information can be provided. It depends 
on many things and should be tested in your specific environment. 
However, I suggest to don't set so more QE without reason.


BR,
Andrei

2/11/2020 9:42 AM, Dorah пишет:

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
   cacheModePARTITIONED
   atomicityModeATOMIC
   backups2
   readFromBackuptrue
   copyOnReadtrue
   statisticsEnabledtrue
   managementEnabledtrue

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

-

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.



But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-



Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issue with large number of Continuous Queries

[zork]

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
  cacheModePARTITIONED
  atomicityModeATOMIC
  backups2
  readFromBackuptrue
  copyOnReadtrue
  statisticsEnabledtrue
  managementEnabledtrue

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

-

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

 

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

 

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issue with large number of Continuous Queries

[zork]

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
  cacheModePARTITIONED
  atomicityModeATOMIC
  backups2
  readFromBackuptrue
  copyOnReadtrue
  statisticsEnabledtrue
  managementEnabledtrue

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

-

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

 

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

 

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issue with large number of Continuous Queries

[Dorah]

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
  cacheModePARTITIONED
  atomicityModeATOMIC
  backups2
  readFromBackuptrue
  copyOnReadtrue
  statisticsEnabledtrue
  managementEnabledtrue

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

-

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

 

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

 

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/