Hi,
Please read some comments below:
1)You said:
*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.
Continues query can handle all updates for some cache. You shouldn't
start special CQ for every entry. Please read the following:
https://apacheignite.readme.io/docs/continuous-queries
2)You said:
With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when a high number of continuous queries machine reaches
100% CPU
utilization.
You can try to monitor what Ignite parts take the CPU and memory usage.
Taking into account the point 1) - you shouldn't have a lot of CQ for
the same cache.
3)You said:
*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.
This is the correct way.
4)You said:
But now the problem is that both the clients do not need to listen to
updates in all the keys in the cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-
Correct you can filter the updates using the following:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/AbstractContinuousQuery.html#getRemoteFilterFactory--
5)You said:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?
I don't think that the current information can be provided. It depends
on many things and should be tested in your specific environment.
However, I suggest to don't set so more QE without reason.
BR,
Andrei
2/11/2020 9:42 AM, Dorah пишет:
*Topology*:
Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
Server-1 running with following jvm params: -Xms1g,-Xmx6g
Client-1 --> Pushing continuous updates on the cache
Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).
The Cache Configuration is as follows:
cacheMode PARTITIONED
atomicityMode ATOMIC
backups 2
readFromBackup true
copyOnRead true
statisticsEnabled true
managementEnabled true
System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.
---------------------------------------------------------
*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.
With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.
*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.
<http://apache-ignite-users.70518.x6.nabble.com/file/t2776/Untitled.png>
But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-
<http://apache-ignite-users.70518.x6.nabble.com/file/t2776/Untitled2.png>
Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?
Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
