Urgent - Using Paired Connections leading to Connection Refuse

[zork]

Hello,

We are facing a critical issue with Ignite 2.8.

We have two machines in same location running different ignite services.
M1 
Ignite server (running on port 48500)
Ignite client-1 (running on port 48100)

M2
Ignite client-2 (running on port 14050)

Port opened between machines:
M2 to M1 - 48500-48520, 48100-48200
M1 to M2 - 14050

Now, M2 is not able to connect to ignite cluster if I set
usePairedConnections to true.
If I have usePairedConnections to false in TCPCommunicationSPI then all
works fine.
 

Here is the related ignite config:
 
  
  





  

  
hkvaspsy-101:48500..48520
  

  

  


  




  

 
 
Connection refused is noticed in the logs on ignite client-2.
 
15:04:06:443908|0327-01043:FFTracer_Ignite [class
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]: {DEBUG} Action {Failed
to join to address [addr=DELVM-M1/10.101.131.171:48500, recon=false,
errs=[java.net.ConnectException: Connection refused (Connection refused)]]}
Thread {tcp-client-disco-msg-worker-#21%WebCluster%}
15:04:06:443975|0296-01043:FFTracer_Ignite [class
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]: {DEBUG} Action {Send
join request [addr=DELVM-M1/10.101.131.171:48501, reconnect=false,
locNodeId=de5d623c-2f6b-4c04-a589-e1812ed1e960]} Thread
{tcp-client-disco-msg-worker-#21%WebCluster%}
15:04:06:444578|1556-01043:FFTracer_Ignite [class
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi]: {ERROR} Action
{Exception on joining: Connection refused (Connection refused)} Thread
{tcp-client-disco-msg-worker-#21%WebCluster%}
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1545)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1507)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:708)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:603)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$1100(ClientImpl.java:141)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:2027)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1683)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:302)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
 
Could you please tell why it is failing in case usePairedConnection is set
to true?
Do we also need 48100-48200 ports opened from M1 to M2 as well?
 
Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance Issue with Enum Serialization in Ignite 2.8

[zork]

Can someone provide inputs on this please?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Performance Issue with Enum Serialization in Ignite 2.8

[zork]

Hello,

I was doing some performance profiling of my ignite.NET client under load
and found that it was taking enormous time while serializing an enum field.
I'm using IBinarizable approach for serialization and my writeBinary method
looks like this:

public void WriteBinary(IBinaryWriter writer)
{
writer.WriteShort(nameof(decimalOffset), decimalOffset);
writer.WriteEnum(nameof(instrumentType), 
instrumentType);  //problematic
writer.WriteObject(nameof(marketKey), marketKey);
writer.WriteString(nameof(marketName), marketName);
writer.WriteEnum(nameof(marketType), marketType);   
//problematic
writer.WriteInt(nameof(priceFormat), priceFormat);
writer.WriteString(nameof(productCode), productCode);
writer.WriteString(nameof(productName), productName);
writer.WriteString(nameof(strategyTemplateName), 
strategyTemplateName);
writer.WriteDouble(nameof(tickSize), tickSize);
}

And my InstrumentType and MarketType enums looks like this:

[IgniteBinaryType] //Order of Enum values has to be identical to
order of Enum values in Java
public enum InstrumentType
{
Undefined,
Stock,
Option,
Future,
Bond,
Strategy,
DNTP,
Forex,
ForexFW,
Repo,
CFD,
TAPO,
CDS,
Swap,
Forward
}

public enum MarketType : byte
{
Undefined = 0,
Outright = 1,
Strategy = 2,
}

I'm also attaching a screenshot of the performance report clearly showing
that it takes around 7.5 secs and 1.3 secs  in 2 WriteEnum() methods while
under 30 miliseconds for the other fields (WriteInt(), WriteString() etc).

WriteEnum.jpg
  

Can someone please suggest what might be causing this and how to workaround
this?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Questions Regarding Critical Workkers Health Check

[zork]

Hello Ignite Experts,

We have recently upgraded from version 2.6 to 2.8.0 and have started to face
some weird behavior since then. 

With the below configuration:
   



We are seeing the below log (with different thread names) multiple times
every second as soon as the ignite server is started:

06-11 08:40:50,309978 [61] ERROR G(Ignite) - Blocked system-critical thread
has been detected. This can lead to cluster-wide undefined behaviour
[workerName=grid-nio-worker-tcp-comm-3,
threadName=grid-nio-worker-tcp-comm-3-#27%IgniteCluster1%, blockedFor=1s]
06-11 08:40:50,310267 [61] WARN  G(Ignite) - Thread
[name="grid-nio-worker-tcp-comm-3-#27%IgniteCluster1%", id=40,
state=RUNNABLE, blockCnt=0, waitCnt=0]

06-11 08:40:50,310609 [61] WARN  (Ignite) - Possible failure suppressed
accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=grid-nio-worker-tcp-comm-3, igniteInstanceName=IgniteCluster1,
finished=false, heartbeatTs=1591864848869]]]
06-11 08:40:50,311975 [61] WARN  CacheDiagnosticManager(Ignite) - Page locks
dump:

Thread=[name=NgServiceProvider_EventsExecutor_0, id=77], state=WAITING
Locked pages = []
Locked pages log: name=NgServiceProvider_EventsExecutor_0
time=(1591864850311, 2020-06-11 14:10:50.311)


Thread=[name=exchange-worker-#43%IgniteCluster1%, id=63],
state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#43%IgniteCluster1%
time=(1591864850311, 2020-06-11 14:10:50.311)


Thread=[name=sys-#45%IgniteCluster1%, id=65], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#45%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)


Thread=[name=sys-#48%IgniteCluster1%, id=68], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#48%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)


Thread=[name=sys-#49%IgniteCluster1%, id=69], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#49%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)


Thread=[name=sys-#51%IgniteCluster1%, id=71], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#51%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)

Full Logs attached here: 
WebIgniteService_WEB.log

  

However, if I change my timeouts like this:




It still occurs but a lot less frequently (I observed it only after I have
added 1 server and 5 clients and the communication started between them).

I did some research and found that this is related to the Critical Workers
Health Check feature which I think is a great addition to ignite but I have
a few questions regarding the same.

Question-1: Does the above log mean that the thread is really blocked? or is
it just busy doing something else?

Question-2: How can I decide the suitable values of these timeouts for my
case? The former values were working for me earlier but now I face this
exception.

Question-3: I can see that this failure type (WORKER_THREAD_BLOCKED) is
actually ignored by default so why do we still see it as an ERROR in logs?

Question-4: As a remedy to this, I have thought of adding another timeout to
my configuration:

I read that failureDetectionTimeout is ignored in case any other timeout is
set. Would that mean now my failure detection timeout would also become
3? or would it mean that failureDetectionTimeout would still be the
configured value and just that it's value will be ignored for
systemWorkerBlockedTimeout (which would now be 3)? 

Question-5: How to decide the value for systemWorkerBlockedTimeout, do we
have some guidelines here?

Question-6: As I can see in
https://issues.apache.org/jira/browse/IGNITE-10154, this
WORKER_THREAD_BLOCKED failure is ignored by default, but on setting some
positive value for systemWorkerBlockedTimeout, it would actually start
working. However, I'm not sure if I want that right now. How else can I
handle this scenario so that I don't get these unnecessary and very frequent
exceptions without enabling this failure? 

Please correct me if I'm wrong anywhere.
Thanks in advance.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Messages being Missed on Node Start

[zork]

Hi,
Can anyone provide an input on this please?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Messages being Missed on Node Start

[zork]

Thanks akorensh,
As advised I'm trying to send a user-defined event from the receiving node
when it becomes ready to accept messages and listening to this event on the
sending node to wait until the receiver is ready.

However, the node that I'm trying to send a user-defined event from is a
.net node.
I have a few questions, but first would like to inquire if it is even
possible to send such an event from a .net node because I'm getting
Ignite-1410 exception when I try to do so. And I can see that IGNITE-1410 is
still open. 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous Query on a varying set of keys

[zork]

Thanks. 
So I can think of two ways using which such a set could be maintained by the
remote node:
1. The remote node listens to a new topic through which the local node sends
it a message whenever the set changes.
2. Or, the local node puts the set values in a new table in the cache itself
and remote node can maybe listen to it using another continuous query.
Is that right?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous Query on a varying set of keys

[zork]

Hi,
Sorry but I could not get it to work.

The standard way of defining a remote filter as shown in sample repo is
something like:

qry.setRemoteFilterFactory(new Factory>() {
@Override public CacheEntryEventFilter
create() {
return new CacheEntryEventFilter()
{
@Override public boolean
evaluate(CacheEntryEvent e) {
return e.getKey() > 10;
}
};
}
});

In the above, instead of having *10* constant, I need to have a HashSet from
which I can check if the updated key exists in it or not (see the snippet
below) And I need the changes in the HashSet to be reflected in the filter.
However it's not making sense to me because the HashSet which is modified is
on one node while the filter is on another node so I expect the remote node
would already have it serialized when the filter was first created and it
would not change even if the set changes in the local node.

HashSet = new HashSet<>();
set.add(20);
qry.setRemoteFilterFactory(new Factory>() {
@Override public CacheEntryEventFilter
create() {
return new CacheEntryEventFilter()
{
@Override public boolean
evaluate(CacheEntryEvent e) {
return set.contains(e.getKey());
}
};
}
});
set.add(10)  // would this actually change the filter on remote node?

Perhaps I'm missing something very obvious here. Please help me identify it.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Messages being Missed on Node Start

[zork]

Hi Ignite experts,

I am facing an issue where some messages sent to a node are sometimes missed
when the node just joins the cluster.

On some debugging, I found that this is because as soon as the node joins
the cluster, the sender node receives a NODE_JOINED event for that receiver
node and it starts sending messages to it, however, the receiver has still
not started listening to those topics which are being received.

Keeping this use case in mind, can someone help answer these please:
1. Can a node register to listen for specific topics before it joins the
cluster?
2. If the above is not possible what would be a good way to achieve this so
that the node that just joined does not miss any messages?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous Query on a varying set of keys

[zork]

Hi Ilya,
Thanks for your response.
I'm aware of remote filters but can these filters be modified once the query
is already attached?
Because if not, then this would not solve my use case as the filter would
always give me updates on a fixed subset of keys, however in my case this
subset is varying (based on what keys a user subscribes from the GUI).



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Continuous Query on a varying set of keys

[zork]

Hi,

We have a table in ignite cache which would have say around 1Mn entries at
anytime. Now we wish to listen on updates on a subset of these keys (say
5-10 thousand keys) and this subset keeps on changing as the user
subscribes/unsubscribes to these keys.

The way it is currently working is one continuous query is attached for
every key whenever it is subscribed and it is closed whenever that key is no
longer of interest (or unsubscribed). The problem with this is that since
there are so many continuous queries (a few thousands), the application goes
out of memory. Also, it would mean all those queries would be evaluated on
the remote node for every update.

To overcome this, what we intend to do is to have just one continuous query
which would listen to all the updates on this table (i.e. all the keys) and
on receiving these updates we would have to filter those of our interest on
our end. But this would mean unnecessary updates would flow over the network
and it doesn't sound like a very good solution too.

Can someone suggest a better way this problem could be addressed? Do we have
something else in ignite to cater such requirement?

Thanks in advance.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issue with large number of Continuous Queries

[zork]

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
  cacheModePARTITIONED
  atomicityModeATOMIC
  backups2
  readFromBackuptrue
  copyOnReadtrue
  statisticsEnabledtrue
  managementEnabledtrue

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

-

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

 

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

 

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issue with large number of Continuous Queries

[zork]

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
  cacheModePARTITIONED
  atomicityModeATOMIC
  backups2
  readFromBackuptrue
  copyOnReadtrue
  statisticsEnabledtrue
  managementEnabledtrue

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

-

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

 

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

 

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/