[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17860091#comment-17860091
]
Pavel Tupitsyn commented on IGNITE-22349:
-
[~kukushal] merged to master:
[a27d21bff3cd5563ab5e3f55cae52df76a917cc8|https://github.com/apache/ignite/commit/a27d21bff3cd5563ab5e3f55cae52df76a917cc8]
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private sealed class PriorityTask : ComputeTaskSplitAdapter bool>
> {
> private readonly int _priority;
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17860075#comment-17860075
]
Alexey Kukushkin commented on IGNITE-22349:
---
[~ptupitsyn], thank you for the veview. Would you be willing to merge the
changes please?
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private sealed class PriorityTask : ComputeTaskSplitAdapter bool>
> {
> private readonly int _priority;
> [TaskSessionResource] private IComputeTaskSession _taskSession;
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859988#comment-17859988
]
Ilya Shishkov commented on IGNITE-22349:
[~kukushal], LGTM.
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private sealed class PriorityTask : ComputeTaskSplitAdapter bool>
> {
> private readonly int _priority;
> [TaskSessionResource] private IComputeTaskSession _taskSession;
> public PriorityTask(int priority)
> {
> _priority =
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859939#comment-17859939
]
Pavel Tupitsyn commented on IGNITE-22349:
-
[~kukushal] looks good to me, thanks!
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private sealed class PriorityTask : ComputeTaskSplitAdapter bool>
> {
> private readonly int _priority;
> [TaskSessionResource] private IComputeTaskSession _taskSession;
> public PriorityTask(int priority)
> {
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859636#comment-17859636
]
Ignite TC Bot commented on IGNITE-22349:
{panel:title=Branch: [pull/11373/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/11373/head] Base: [master] : New Tests
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Platform .NET (Core Linux){color} [[tests
2|https://ci2.ignite.apache.org/viewLog.html?buildId=7927135]]
* {color:#013220}DotNetCore:
ComputeTaskSessionTest.DistributesTaskSessionAttributeLocally - PASSED{color}
* {color:#013220}DotNetCore:
ComputeTaskSessionTest.DistributesTaskSessionAttributeRemotely - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7927170buildTypeId=IgniteTests24Java8_RunAll]
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859584#comment-17859584
]
Alexey Kukushkin commented on IGNITE-22349:
---
[~ptupitsyn], [~shishkovilja], I addressed your code review comments and also
fixed some C++ code that I forgot to initially change and that caused some C++
test failures. Could you please take another look and approve the PR if you
have no other comments. Thank you!
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859578#comment-17859578
]
Ignite TC Bot commented on IGNITE-22349:
{panel:title=Branch: [pull/11373/head] Base: [master] : Possible Blockers
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}PDS 4{color} [[tests
2|https://ci2.ignite.apache.org/viewLog.html?buildId=7927124]]
* IgnitePdsTestSuite4:
IgniteClusterActivateDeactivateTestWithPersistenceAndMemoryReuse.testActivateReadOnlyCachesRestore_SingleNode
- Test has low fail rate in base branch 0,0% and is not flaky
* IgnitePdsTestSuite4:
IgniteClusterActivateDeactivateTestWithPersistenceAndMemoryReuse.testInactiveTopologyChangesReadOnly
- Test has low fail rate in base branch 0,0% and is not flaky
{panel}
{panel:title=Branch: [pull/11373/head] Base: [master] : New Tests
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Platform .NET (Core Linux){color} [[tests
2|https://ci2.ignite.apache.org/viewLog.html?buildId=7927135]]
* {color:#013220}DotNetCore:
ComputeTaskSessionTest.DistributesTaskSessionAttributeLocally - PASSED{color}
* {color:#013220}DotNetCore:
ComputeTaskSessionTest.DistributesTaskSessionAttributeRemotely - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7927170buildTypeId=IgniteTests24Java8_RunAll]
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17852287#comment-17852287
]
Alexey Kukushkin commented on IGNITE-22349:
---
[~ptupitsyn], thank you for volunteering to review the changes and for the
comments. I will address your comments and I am also running the tests now to
make sure nothing is broken. I will let you know when everything is done.
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Attachments: IGNITE-22349-README.md, ignite-22349.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17852282#comment-17852282
]
Pavel Tupitsyn commented on IGNITE-22349:
-
[~kukushal] Looks good to me in general, please see a few minor comments on
GitHub.
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Attachments: IGNITE-22349-README.md, ignite-22349.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private sealed class PriorityTask : ComputeTaskSplitAdapter bool>
> {
> private readonly int _priority;
>
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[
https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851606#comment-17851606
]
Pavel Tupitsyn commented on IGNITE-22349:
-
[~kukushal] I would like to do a review, do you prefer to create a PR for that,
or use JIRA comments?
> Ignite.NET support for priority ordering of Compute jobs
>
>
> Key: IGNITE-22349
> URL: https://issues.apache.org/jira/browse/IGNITE-22349
> Project: Ignite
> Issue Type: Improvement
> Components: .NET, compute, platforms
>Affects Versions: 2.16
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
> Labels: .net, compute, ise, platforms
> Attachments: IGNITE-22349-README.md, ignite-22349.patch
>
>
> I want Apache Ignite to support [priority
> ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering]
> of Ignite.NET compute jobs on the same node.
> h4. Analysis
> {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the
> {{PriorityQueueCollisionSpi}} expects the user to provide job priorities
> via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the
> {{ComputeTaskSession}} is not available in Ignite.NET.
> It looks like the requirement is to add an injectable {{ComputeTaskSession}}
> in Ignite.NET exposing the {{SetAttributes}} operation similar to how it
> works in Java.
> h4. Reproducer
> I expect more or less ordered output from the below reproducer. The output
> may not be completely ordered since completely ordered output requires all
> the jobs to land on the server node in single batch and this reproducer
> cannot guarantee that:
> {noformat}
> >>> Completed job with priority 0
> >>> Completed job with priority 9
> >>> Completed job with priority 8
> >>> Completed job with priority 7
> >>> Completed job with priority 6
> >>> Completed job with priority 5
> >>> Completed job with priority 4
> >>> Completed job with priority 3
> >>> Completed job with priority 2
> >>> Completed job with priority 1
> {noformat}
> {{PriorityQueueCollisionSpiTest.cs}}:
> {code:java}
> public class PriorityQueueCollisionSpiTest
> {
> private static ITestOutputHelper? _output;
> public PriorityQueueCollisionSpiTest(ITestOutputHelper output)
> {
> _output = output;
> }
> ///
> /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority.
> ///
> [Fact]
> public void SchedulesJobsAccordingToTaskPriority()
> {
> // Given an Ignite cluster consisting of server and client nodes
> using var ignored = Ignition.Start(GetIgniteConfiguration("server1"));
> var igniteConfiguration = GetIgniteConfiguration("app1");
> igniteConfiguration.ClientMode = true;
> using var ignite = Ignition.Start(igniteConfiguration);
> var igniteCompute = ignite.GetCompute();
>
> // And the user asynchronously executes multiple tasks, each task
> starting a job having increasing priority
> const int jobCount = 10;
> ICollection futureResultCollection = new List(jobCount);
> for (var priority = 0; priority < jobCount; priority++)
> {
> var task = new PriorityTask(priority);
> var futureResult = igniteCompute.ExecuteAsync(task, jobCount);
> futureResultCollection.Add(futureResult);
> }
> // When all the jobs complete
> Task.WaitAll(futureResultCollection.ToArray());
>
> // Then the ">>> Completed job with priority" console output
> demonstrates that the jobs completed in the
> // decreasing priority order, more or less.
> }
> private static IgniteConfiguration GetIgniteConfiguration(string
> igniteName) =>
> new()
> {
> ConsistentId = igniteName,
> IgniteInstanceName = igniteName,
> SpringConfigUrl = "ignite-sandbox.xml",
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new
> List {"127.0.0.1:48700"}},
> LocalPort = 48700
> },
> FailureDetectionTimeout = TimeSpan.FromMinutes(10),
> ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10),
> JvmOptions = new List
> {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"}
> };
> ///
> /// implementation that
> single s with
> /// the specified priority.
> ///
> [ComputeTaskSessionFullSupport]
> private sealed class PriorityTask : ComputeTaskSplitAdapter bool>
> {
> private readonly int _priority;
> [TaskSessionResource] private
