Hi Adrian,
One thing I would be careful about in .Net IA clients is managing the
numbers of allocations. Your test suggests the compute func being sent to
the node contains a large number of objects (10 in the case of your
test scaffold).
Increasing the size of the public thread pool will increase the number of
concurrent tasks being executed, also increasing the number of objects on
the code. If you had 48 concurrent tasks at 10 objects each, that is
4.8 million objects.
You mention the runtime for the small number of threads is already in the
minutes, so the GC will may have promoted large numbers of objects into the
Gen 2 pool which means every GC epoch must scan large numbers of objects.
I would look at your GC pause lengths as an initial possibility. I would
also look at structuring the payload so that it does not cause large
numbers of allocations in the processing context.
Beyond that, I would look at any locking your code may be performing in its
processing, and how individual compute tasks make use of the CPU/mem on the
server. C# Tasks and async programming may also be a good option to look at.
Cheers,
Raymond.
On Tue, Apr 27, 2021 at 8:36 AM Adrian Corman
wrote:
> Hi Igniters,
>
>
>
> We are using the .net API for Ignite to distribute computing tasks
> (closures) to a cluster. We are having an issue involving the public thread
> pool size – when the tasks are sent to a machine with the default public
> thread pool size (48 processors, 48 threads in the case of our test
> machine), the jobs run 10x slower than if we set the public thread pool to
> a smaller number such as 6 or 12. This slowdown doesn’t seem to be related
> to any cache operations or IO – logging shows that even pure computational
> methods are slower overall.
>
>
>
> We also discovered that when we run two nodes on the test machine, each
> with half the thread pool size, the jobs run faster than with one node with
> the full pool size (two of 24 run faster than one of 48).
>
>
>
> Does anyone have any suggestions on how we can trace the source of this
> slowdown, or otherwise improve performance? I’ve included scaffolding for
> the types of tests we are running, to illustrate our issue.
>
>
>
> Thank you!
>
>
>
> public class PerfTest_Ignite
>
> {
>
> private IIgnite ignite;
>
>
>
> public void Test_PublicThreadPool_12()
>
> {
>
> ignite = Ignition.Start(new IgniteConfiguration
>
> {
>
> PublicThreadPoolSize = 12
>
> });
>
> string[] tasks = new string[10];
>
>
>
> var result = ignite.GetCompute().ApplyAsync(new
> LongRunning_DotNet_Task(), tasks);
>
> // This method performs as expected
>
> }
>
>
>
> public void Test_PublicThreadPool_48()
>
> {
>
> ignite = Ignition.Start(new IgniteConfiguration
>
> {
>
> //PublicThreadPoolSize = 48. Do not set Public Thread Pool
> size. (48 is the number of cores on this machine. )
>
> });
>
> string[] tasks = new string[10];
>
>
>
> var result = ignite.GetCompute().ApplyAsync(new
> LongRunning_DotNet_Task(), tasks);
>
>
>
> // This method is 10 TIMES SLOWER?
>
> }
>
>
>
>
>
> }
>
>
>
>
>
> public class LongRunning_DotNet_Task : IComputeFunc
>
> {
>
> public bool Invoke(string[] arg)
>
> {
>
> // do work that takes up to 5 minutes with 12 threads, but
> takes 50 minutes with 48 threads!
>
> return true;
>
>}
>
> }
>
>
>
--
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wil...@trimble.com
