I am testing an aspect of a POC I have written in C# to determine how well
it performs when processing multiple tasks at one time. I’m using Ignite
2.2.
Briefly, a client node sends a request to a server node for a list of
items. Each of the items requested is send back to a listener in the client
(using Ignite messaging) and then processed (rendered onto a bitmap). Once
all parts of the request have arrived and been rendered, a wait state is
triggered allowing the request to be completed. The wait state will
terminate after 2 minutes which short cuts the request in the client node.
This works well from a functional perspective, but I ran into problems with
multithreading.
To test multi-threading, I essentially did this:
int nThreads = 4;
Parallel.For(0, nThreads, x => PerformRender(<some arguments>));
This works well for values of nThreads of up to 7. However, once nThreads
is set more than 7 then the requests start stalling, and no progress is
made the first eight requests until the wait states start timing out at two
minutes.
Looking in the log is it common to see entries like this during the time
the requests are stalled:
WARN 2017-10-04 13:41:57,051 252758ms
IgniteKernal%Raptor ? - Possible
thread pool starvation detected (no task completed in last 30000ms, is
public thread pool size large enough?)
Here we see an internal monitor is warning regarding a possible deadlock or
race condition meaning progress is not being made on requests.
INFO 2017-10-04 15:47:21,182 263409ms
IgniteKernal%Raptor ? -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=c966b7cc, name=Raptor, uptime=00:04:10:105]
^-- H/N/C [hosts=1, nodes=7, CPUs=8]
^-- CPU [cur=0.07%, avg=1.76%, GC=0%]
^-- PageMemory [pages=0]
^-- Heap [used=103MB, free=88.58%, comm=497MB]
^-- Non heap [used=38MB, free=-1%, comm=39MB]
^-- Public thread pool [active=8, idle=0, qSize=274]
^-- System thread pool [active=0, idle=0, qSize=0]
^-- Outbound messages queue [size=0]
Here we see the pubic thread pool is set to 8 by default. This suggests
when the pool becomes saturated with requests something is stalling the
requests.
I tried increasing the number of threads in the public pool in the grid
configuration and this resulted in no stalls in requests which suggests
Ignite does not like fully committed thread pools.
Is this a known issue with Ignite?
Thanks,
Raymond.
