[jira] [Comment Edited] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860075#comment-17860075 ] Alexey Kukushkin edited comment on IGNITE-22349 at 6/26/24 6:31 AM: [~ptupitsyn], thank you for the review. Would you be willing to merge the changes please? was (Author: kukushal): [~ptupitsyn], thank you for the veview. Would you be willing to merge the changes please? > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Time Spent: 50m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > private sea
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860075#comment-17860075 ] Alexey Kukushkin commented on IGNITE-22349: --- [~ptupitsyn], thank you for the veview. Would you be willing to merge the changes please? > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Time Spent: 50m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > private sealed class PriorityTask : ComputeTaskSplitAdapter bool> > { > private readonly int _priority; > [TaskSessionResource] private IComputeTaskSession _
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17859584#comment-17859584 ] Alexey Kukushkin commented on IGNITE-22349: --- [~ptupitsyn], [~shishkovilja], I addressed your code review comments and also fixed some C++ code that I forgot to initially change and that caused some C++ test failures. Could you please take another look and approve the PR if you have no other comments. Thank you! > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Time Spent: 20m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] >
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Attachment: (was: IGNITE-22349-README.md) > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Time Spent: 20m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > private sealed class PriorityTask : ComputeTaskSplitAdapter bool> > { > private readonly int _priority; > [TaskSessionResource] private IComputeTaskSession _taskSession; > public PriorityTask(int priority) > { > _priority = prio
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Attachment: (was: ignite-22349.patch) > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Time Spent: 20m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > private sealed class PriorityTask : ComputeTaskSplitAdapter bool> > { > private readonly int _priority; > [TaskSessionResource] private IComputeTaskSession _taskSession; > public PriorityTask(int priority) > { > _priority = priority
[jira] [Commented] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852287#comment-17852287 ] Alexey Kukushkin commented on IGNITE-22349: --- [~ptupitsyn], thank you for volunteering to review the changes and for the comments. I will address your comments and I am also running the tests now to make sure nothing is broken. I will let you know when everything is done. > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Attachments: IGNITE-22349-README.md, ignite-22349.patch > > Time Spent: 20m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeT
[jira] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349 ] Alexey Kukushkin deleted comment on IGNITE-22349: --- was (Author: kukushal): h3. Patch * Overview: [^IGNITE-22349-README.md] * Diff with tag 2.16.0: [^ignite-22349.patch] > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, ise, platforms > Attachments: IGNITE-22349-README.md, ignite-22349.patch > > Time Spent: 10m > Remaining Estimate: 0h > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > private sealed class PriorityTask : ComputeTaskSplitAdapter bool> > { > private readonly int _priority; > [TaskSessionResource] private IComputeTaskSession _ta
[jira] [Comment Edited] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850271#comment-17850271 ] Alexey Kukushkin edited comment on IGNITE-22349 at 5/29/24 6:39 AM: h3. Patch * Overview: [^IGNITE-22349-README.md] * Diff with tag 2.16.0: [^ignite-22349.patch] was (Author: kukushal): h3. Patch * Overview: [^IGNITE-22349-README.md] * Diff: [^ignite-22349.patch] > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, platforms > Attachments: IGNITE-22349-README.md, ignite-22349.patch > > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > pri
[jira] [Comment Edited] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850271#comment-17850271 ] Alexey Kukushkin edited comment on IGNITE-22349 at 5/29/24 6:38 AM: h3. Patch * Overview: [^IGNITE-22349-README.md] * Diff: [^ignite-22349.patch] was (Author: kukushal): h3. Patch * Overview: attached IGNITE-22349-README.md * Diff: attached ignite-22349.patch > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, platforms > Attachments: IGNITE-22349-README.md, ignite-22349.patch > > > I want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h4. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to how it > works in Java. > h4. Reproducer > I expect more or less ordered output from the below reproducer. The output > may not be completely ordered since completely ordered output requires all > the jobs to land on the server node in single batch and this reproducer > cannot guarantee that: > {noformat} > >>> Completed job with priority 0 > >>> Completed job with priority 9 > >>> Completed job with priority 8 > >>> Completed job with priority 7 > >>> Completed job with priority 6 > >>> Completed job with priority 5 > >>> Completed job with priority 4 > >>> Completed job with priority 3 > >>> Completed job with priority 2 > >>> Completed job with priority 1 > {noformat} > {{PriorityQueueCollisionSpiTest.cs}}: > {code:java} > public class PriorityQueueCollisionSpiTest > { > private static ITestOutputHelper? _output; > public PriorityQueueCollisionSpiTest(ITestOutputHelper output) > { > _output = output; > } > /// > /// Schedules jobs according to cref="IComputeTask{TArg,TJobRes,TRes}"/>'s priority. > /// > [Fact] > public void SchedulesJobsAccordingToTaskPriority() > { > // Given an Ignite cluster consisting of server and client nodes > using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); > var igniteConfiguration = GetIgniteConfiguration("app1"); > igniteConfiguration.ClientMode = true; > using var ignite = Ignition.Start(igniteConfiguration); > var igniteCompute = ignite.GetCompute(); > > // And the user asynchronously executes multiple tasks, each task > starting a job having increasing priority > const int jobCount = 10; > ICollection futureResultCollection = new List(jobCount); > for (var priority = 0; priority < jobCount; priority++) > { > var task = new PriorityTask(priority); > var futureResult = igniteCompute.ExecuteAsync(task, jobCount); > futureResultCollection.Add(futureResult); > } > // When all the jobs complete > Task.WaitAll(futureResultCollection.ToArray()); > > // Then the ">>> Completed job with priority" console output > demonstrates that the jobs completed in the > // decreasing priority order, more or less. > } > private static IgniteConfiguration GetIgniteConfiguration(string > igniteName) => > new() > { > ConsistentId = igniteName, > IgniteInstanceName = igniteName, > SpringConfigUrl = "ignite-sandbox.xml", > DiscoverySpi = new TcpDiscoverySpi > { > IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new > List {"127.0.0.1:48700"}}, > LocalPort = 48700 > }, > FailureDetectionTimeout = TimeSpan.FromMinutes(10), > ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), > JvmOptions = new List > {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} > }; > /// > /// implementation that > single s with > /// the specified priority. > /// > [ComputeTaskSessionFullSupport] > private sea
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: I want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h4. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to how it works in Java. h4. Reproducer I expect more or less ordered output from the below reproducer. The output may not be completely ordered since completely ordered output requires all the jobs to land on the server node in single batch and this reproducer cannot guarantee that: {noformat} >>> Completed job with priority 0 >>> Completed job with priority 9 >>> Completed job with priority 8 >>> Completed job with priority 7 >>> Completed job with priority 6 >>> Completed job with priority 5 >>> Completed job with priority 4 >>> Completed job with priority 3 >>> Completed job with priority 2 >>> Completed job with priority 1 {noformat} {{PriorityQueueCollisionSpiTest.cs}}: {code:java} public class PriorityQueueCollisionSpiTest { private static ITestOutputHelper? _output; public PriorityQueueCollisionSpiTest(ITestOutputHelper output) { _output = output; } /// /// Schedules jobs according to 's priority. /// [Fact] public void SchedulesJobsAccordingToTaskPriority() { // Given an Ignite cluster consisting of server and client nodes using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); var igniteConfiguration = GetIgniteConfiguration("app1"); igniteConfiguration.ClientMode = true; using var ignite = Ignition.Start(igniteConfiguration); var igniteCompute = ignite.GetCompute(); // And the user asynchronously executes multiple tasks, each task starting a job having increasing priority const int jobCount = 10; ICollection futureResultCollection = new List(jobCount); for (var priority = 0; priority < jobCount; priority++) { var task = new PriorityTask(priority); var futureResult = igniteCompute.ExecuteAsync(task, jobCount); futureResultCollection.Add(futureResult); } // When all the jobs complete Task.WaitAll(futureResultCollection.ToArray()); // Then the ">>> Completed job with priority" console output demonstrates that the jobs completed in the // decreasing priority order, more or less. } private static IgniteConfiguration GetIgniteConfiguration(string igniteName) => new() { ConsistentId = igniteName, IgniteInstanceName = igniteName, SpringConfigUrl = "ignite-sandbox.xml", DiscoverySpi = new TcpDiscoverySpi { IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new List {"127.0.0.1:48700"}}, LocalPort = 48700 }, FailureDetectionTimeout = TimeSpan.FromMinutes(10), ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), JvmOptions = new List {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} }; /// /// implementation that single s with /// the specified priority. /// [ComputeTaskSessionFullSupport] private sealed class PriorityTask : ComputeTaskSplitAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; public PriorityTask(int priority) { _priority = priority; } /// public override bool Reduce(IList> results) => results.All(r => r.Data); /// protected override ICollection> Split(int gridSize, int jobCount) { IComputeJob job = new PriorityJob(_priority); _taskSession.SetAttribute("grid.task.priority", _priority); var actual = _taskSession.GetAttribute("grid.task.priority"); Assert.Equal(_priority, actual); return new List> {job}; } } /// /// implementation with a priority indicator. /// private class PriorityJob : ComputeJobAdapter { private readonly int _priority; public PriorityJob(int priority) { _priority = priority; } /// public override bool Execute() { _o
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: I want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h4. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to how it works in Java. h4. Reproducer I expect more or less ordered output from the below reproducer. The output may not be completely ordered since completely ordered output requires all the jobs to land on the server node in single batch and this reproducer cannot guarantee that: {noformat} >>> Completed job with priority 0 >>> Completed job with priority 9 >>> Completed job with priority 8 >>> Completed job with priority 7 >>> Completed job with priority 6 >>> Completed job with priority 5 >>> Completed job with priority 4 >>> Completed job with priority 3 >>> Completed job with priority 2 >>> Completed job with priority 1 {noformat} {{PriorityQueueCollisionSpiTest.cs}}: {code:java} public class PriorityQueueCollisionSpiTest { private static ITestOutputHelper? _output; public PriorityQueueCollisionSpiTest(ITestOutputHelper output) { _output = output; } /// /// Schedules jobs according to 's priority. /// [Fact] public void SchedulesJobsAccordingToTaskPriority() { // Given an Ignite cluster consisting of server and client nodes using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); var igniteConfiguration = GetIgniteConfiguration("app1"); igniteConfiguration.ClientMode = true; using var ignite = Ignition.Start(igniteConfiguration); var igniteCompute = ignite.GetCompute(); // And the user asynchronously executes multiple tasks, each task starting a job having increasing priority const int jobCount = 10; ICollection futureResultCollection = new List(jobCount); for (var priority = 0; priority < jobCount; priority++) { var task = new PriorityTask(priority); var futureResult = igniteCompute.ExecuteAsync(task, jobCount); futureResultCollection.Add(futureResult); } // When all the jobs complete Task.WaitAll(futureResultCollection.ToArray()); // Then the ">>> Completed job with priority" console output demonstrates that the jobs completed in the // decreasing priority order, more or less. } private static IgniteConfiguration GetIgniteConfiguration(string igniteName) => new() { ConsistentId = igniteName, IgniteInstanceName = igniteName, SpringConfigUrl = "ignite-sandbox.xml", DiscoverySpi = new TcpDiscoverySpi { IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new List {"127.0.0.1:48700"}}, LocalPort = 48700 }, FailureDetectionTimeout = TimeSpan.FromMinutes(10), ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), JvmOptions = new List {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} }; /// /// implementation that single s with /// the specified priority. /// [ComputeTaskSessionFullSupport] private sealed class PriorityTask : ComputeTaskSplitAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityTask(int priority) { _priority = priority; } /// public override bool Reduce(IList> results) => results.All(r => r.Data); /// protected override ICollection> Split(int gridSize, int jobCount) { IComputeJob job = new PriorityJob(_priority); _taskSession.SetAttribute("grid.task.priority", _priority); var actual = _taskSession.GetAttribute("grid.task.priority"); Assert.Equal(_priority, actual); return new List> {job}; } } /// /// implementation with a priority indicator. /// private class PriorityJob : ComputeJobAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignit
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: I want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h4. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. h4. Reproducer I expect more or less ordered output from the below reproducer. The output may not be completely ordered since completely ordered output requires all the jobs to land on the server node in single batch and this reproducer cannot guarantee that: {noformat} >>> Completed job with priority 0 >>> Completed job with priority 9 >>> Completed job with priority 8 >>> Completed job with priority 7 >>> Completed job with priority 6 >>> Completed job with priority 5 >>> Completed job with priority 4 >>> Completed job with priority 3 >>> Completed job with priority 2 >>> Completed job with priority 1 {noformat} {{PriorityQueueCollisionSpiTest.cs}}: {code:java} public class PriorityQueueCollisionSpiTest { private static ITestOutputHelper? _output; public PriorityQueueCollisionSpiTest(ITestOutputHelper output) { _output = output; } /// /// Schedules jobs according to 's priority. /// [Fact] public void SchedulesJobsAccordingToTaskPriority() { // Given an Ignite cluster consisting of server and client nodes using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); var igniteConfiguration = GetIgniteConfiguration("app1"); igniteConfiguration.ClientMode = true; using var ignite = Ignition.Start(igniteConfiguration); var igniteCompute = ignite.GetCompute(); // And the user asynchronously executes multiple tasks, each task starting a job having increasing priority const int jobCount = 10; ICollection futureResultCollection = new List(jobCount); for (var priority = 0; priority < jobCount; priority++) { var task = new PriorityTask(priority); var futureResult = igniteCompute.ExecuteAsync(task, jobCount); futureResultCollection.Add(futureResult); } // When all the jobs complete Task.WaitAll(futureResultCollection.ToArray()); // Then the ">>> Completed job with priority" console output demonstrates that the jobs completed in the // decreasing priority order, more or less. } private static IgniteConfiguration GetIgniteConfiguration(string igniteName) => new() { ConsistentId = igniteName, IgniteInstanceName = igniteName, SpringConfigUrl = "ignite-sandbox.xml", DiscoverySpi = new TcpDiscoverySpi { IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new List {"127.0.0.1:48700"}}, LocalPort = 48700 }, FailureDetectionTimeout = TimeSpan.FromMinutes(10), ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), JvmOptions = new List {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} }; /// /// implementation that single s with /// the specified priority. /// [ComputeTaskSessionFullSupport] private sealed class PriorityTask : ComputeTaskSplitAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityTask(int priority) { _priority = priority; } /// public override bool Reduce(IList> results) => results.All(r => r.Data); /// protected override ICollection> Split(int gridSize, int jobCount) { IComputeJob job = new PriorityJob(_priority); _taskSession.SetAttribute("grid.task.priority", _priority); var actual = _taskSession.GetAttribute("grid.task.priority"); Assert.Equal(_priority, actual); return new List> {job}; } } /// /// implementation with a priority indicator. /// private class PriorityJob : ComputeJobAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: We want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h4. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. h4. Reproducer We expect more or less ordered output from the below reproducer. The output may not be completely ordered since completely ordered output requires all the jobs to land on the server node in single batch and this reproducer cannot guarantee that: {noformat} >>> Completed job with priority 0 >>> Completed job with priority 9 >>> Completed job with priority 8 >>> Completed job with priority 7 >>> Completed job with priority 6 >>> Completed job with priority 5 >>> Completed job with priority 4 >>> Completed job with priority 3 >>> Completed job with priority 2 >>> Completed job with priority 1 {noformat} {{PriorityQueueCollisionSpiTest.cs}}: {code:java} public class PriorityQueueCollisionSpiTest { private static ITestOutputHelper? _output; public PriorityQueueCollisionSpiTest(ITestOutputHelper output) { _output = output; } /// /// Schedules jobs according to 's priority. /// [Fact] public void SchedulesJobsAccordingToTaskPriority() { // Given an Ignite cluster consisting of server and client nodes using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); var igniteConfiguration = GetIgniteConfiguration("app1"); igniteConfiguration.ClientMode = true; using var ignite = Ignition.Start(igniteConfiguration); var igniteCompute = ignite.GetCompute(); // And the user asynchronously executes multiple tasks, each task starting a job having increasing priority const int jobCount = 10; ICollection futureResultCollection = new List(jobCount); for (var priority = 0; priority < jobCount; priority++) { var task = new PriorityTask(priority); var futureResult = igniteCompute.ExecuteAsync(task, jobCount); futureResultCollection.Add(futureResult); } // When all the jobs complete Task.WaitAll(futureResultCollection.ToArray()); // Then the ">>> Completed job with priority" console output demonstrates that the jobs completed in the // decreasing priority order, more or less. } private static IgniteConfiguration GetIgniteConfiguration(string igniteName) => new() { ConsistentId = igniteName, IgniteInstanceName = igniteName, SpringConfigUrl = "ignite-sandbox.xml", DiscoverySpi = new TcpDiscoverySpi { IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new List {"127.0.0.1:48700"}}, LocalPort = 48700 }, FailureDetectionTimeout = TimeSpan.FromMinutes(10), ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), JvmOptions = new List {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} }; /// /// implementation that single s with /// the specified priority. /// [ComputeTaskSessionFullSupport] private sealed class PriorityTask : ComputeTaskSplitAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityTask(int priority) { _priority = priority; } /// public override bool Reduce(IList> results) => results.All(r => r.Data); /// protected override ICollection> Split(int gridSize, int jobCount) { IComputeJob job = new PriorityJob(_priority); _taskSession.SetAttribute("grid.task.priority", _priority); var actual = _taskSession.GetAttribute("grid.task.priority"); Assert.Equal(_priority, actual); return new List> {job}; } } /// /// implementation with a priority indicator. /// private class PriorityJob : ComputeJobAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResour
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: We want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h4. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. h4. Reproducer We expect more or less ordered output from the below reproducer. The output may not be completely ordered since completely ordered output requires all the jobs to land on the server node in single batch and this reproducer cannot guarantee that: {noformat} >>> Completed job with priority 0 >>> Completed job with priority 9 >>> Completed job with priority 8 >>> Completed job with priority 7 >>> Completed job with priority 6 >>> Completed job with priority 5 >>> Completed job with priority 4 >>> Completed job with priority 3 >>> Completed job with priority 2 >>> Completed job with priority 1 {noformat} {{PriorityQueueCollisionSpiTest.cs}}: {code:java} public class PriorityQueueCollisionSpiTest { private static ITestOutputHelper? _output; /// /// Schedules jobs according to 's priority. /// [Fact] public void SchedulesJobsAccordingToTaskPriority() { // Given an Ignite cluster consisting of server and client nodes using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); var igniteConfiguration = GetIgniteConfiguration("app1"); igniteConfiguration.ClientMode = true; using var ignite = Ignition.Start(igniteConfiguration); var igniteCompute = ignite.GetCompute(); // And the user asynchronously executes multiple tasks, each task starting a job having increasing priority const int jobCount = 10; ICollection futureResultCollection = new List(jobCount); for (var priority = 0; priority < jobCount; priority++) { var task = new PriorityTask(priority); var futureResult = igniteCompute.ExecuteAsync(task, jobCount); futureResultCollection.Add(futureResult); } // When all the jobs complete Task.WaitAll(futureResultCollection.ToArray()); // Then the ">>> Completed job with priority" console output demonstrates that the jobs completed in the // decreasing priority order, more or less. } private static IgniteConfiguration GetIgniteConfiguration(string igniteName) => new() { ConsistentId = igniteName, IgniteInstanceName = igniteName, SpringConfigUrl = "ignite-sandbox.xml", DiscoverySpi = new TcpDiscoverySpi { IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new List {"127.0.0.1:48700"}}, LocalPort = 48700 }, FailureDetectionTimeout = TimeSpan.FromMinutes(10), ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), JvmOptions = new List {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} }; /// /// implementation that single s with /// the specified priority. /// [ComputeTaskSessionFullSupport] private sealed class PriorityTask : ComputeTaskSplitAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityTask(int priority) { _priority = priority; } /// public override bool Reduce(IList> results) => results.All(r => r.Data); /// protected override ICollection> Split(int gridSize, int jobCount) { IComputeJob job = new PriorityJob(_priority); _taskSession.SetAttribute("grid.task.priority", _priority); var actual = _taskSession.GetAttribute("grid.task.priority"); Assert.Equal(_priority, actual); return new List> {job}; } } /// /// implementation with a priority indicator. /// private class PriorityJob : ComputeJobAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityJob(int priority) { _priority = pr
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: We want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h4. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. h4. Reproducer We expect more or less ordered output from the below reproducer. The output may not be completely ordered since this reproducer does not guarantee : {noformat} >>> Completed job with priority 0 >>> Completed job with priority 9 >>> Completed job with priority 8 >>> Completed job with priority 7 >>> Completed job with priority 6 >>> Completed job with priority 5 >>> Completed job with priority 4 >>> Completed job with priority 3 >>> Completed job with priority 2 >>> Completed job with priority 1 {noformat} {{PriorityQueueCollisionSpiTest.cs}}: {code:java} public class PriorityQueueCollisionSpiTest { private static ITestOutputHelper? _output; /// /// Schedules jobs according to 's priority. /// [Fact] public void SchedulesJobsAccordingToTaskPriority() { // Given an Ignite cluster consisting of server and client nodes using var ignored = Ignition.Start(GetIgniteConfiguration("server1")); var igniteConfiguration = GetIgniteConfiguration("app1"); igniteConfiguration.ClientMode = true; using var ignite = Ignition.Start(igniteConfiguration); var igniteCompute = ignite.GetCompute(); // And the user asynchronously executes multiple tasks, each task starting a job having increasing priority const int jobCount = 10; ICollection futureResultCollection = new List(jobCount); for (var priority = 0; priority < jobCount; priority++) { var task = new PriorityTask(priority); var futureResult = igniteCompute.ExecuteAsync(task, jobCount); futureResultCollection.Add(futureResult); } // When all the jobs complete Task.WaitAll(futureResultCollection.ToArray()); // Then the ">>> Completed job with priority" console output demonstrates that the jobs completed in the // decreasing priority order, more or less. } private static IgniteConfiguration GetIgniteConfiguration(string igniteName) => new() { ConsistentId = igniteName, IgniteInstanceName = igniteName, SpringConfigUrl = "ignite-sandbox.xml", DiscoverySpi = new TcpDiscoverySpi { IpFinder = new TcpDiscoveryStaticIpFinder {Endpoints = new List {"127.0.0.1:48700"}}, LocalPort = 48700 }, FailureDetectionTimeout = TimeSpan.FromMinutes(10), ClientFailureDetectionTimeout = TimeSpan.FromMinutes(10), JvmOptions = new List {"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"} }; /// /// implementation that single s with /// the specified priority. /// [ComputeTaskSessionFullSupport] private sealed class PriorityTask : ComputeTaskSplitAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityTask(int priority) { _priority = priority; } /// public override bool Reduce(IList> results) => results.All(r => r.Data); /// protected override ICollection> Split(int gridSize, int jobCount) { IComputeJob job = new PriorityJob(_priority); _taskSession.SetAttribute("grid.task.priority", _priority); var actual = _taskSession.GetAttribute("grid.task.priority"); Assert.Equal(_priority, actual); return new List> {job}; } } /// /// implementation with a priority indicator. /// private class PriorityJob : ComputeJobAdapter { private readonly int _priority; [TaskSessionResource] private IComputeTaskSession _taskSession; [InstanceResource] private IIgnite _ignite; public PriorityJob(int priority) { _priority = priority; } /// public override bool Execute() { _outpu
[jira] [Updated] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
[ https://issues.apache.org/jira/browse/IGNITE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-22349: -- Description: We want Apache Ignite to support [priority ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] of Ignite.NET compute jobs on the same node. h3. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. was: We want Apache Ignite to support [priority ordering](https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering) of Ignite.NET compute jobs on the same node. h3. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. > Ignite.NET support for priority ordering of Compute jobs > > > Key: IGNITE-22349 > URL: https://issues.apache.org/jira/browse/IGNITE-22349 > Project: Ignite > Issue Type: Improvement > Components: .NET, compute, platforms >Affects Versions: 2.16 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: .net, compute, platforms > > We want Apache Ignite to support [priority > ordering|https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering] > of Ignite.NET compute jobs on the same node. > h3. Analysis > {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the > {{PriorityQueueCollisionSpi}} expects the user to provide job priorities > via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the > {{ComputeTaskSession}} is not available in Ignite.NET. > It looks like the requirement is to add an injectable {{ComputeTaskSession}} > in Ignite.NET exposing the {{SetAttributes}} operation similar to what the > {{ComputeTaskSession}} does in Java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-22349) Ignite.NET support for priority ordering of Compute jobs
Alexey Kukushkin created IGNITE-22349: - Summary: Ignite.NET support for priority ordering of Compute jobs Key: IGNITE-22349 URL: https://issues.apache.org/jira/browse/IGNITE-22349 Project: Ignite Issue Type: Improvement Components: .NET, compute, platforms Affects Versions: 2.16 Reporter: Alexey Kukushkin Assignee: Alexey Kukushkin We want Apache Ignite to support [priority ordering](https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling#priority-ordering) of Ignite.NET compute jobs on the same node. h3. Analysis {{PriorityQueueCollisionSpi}} does priority ordering. The problem is the {{PriorityQueueCollisionSpi}} expects the user to provide job priorities via the {{ComputeTaskSession}}'s "{{grid.task.priority}}" attribute and the {{ComputeTaskSession}} is not available in Ignite.NET. It looks like the requirement is to add an injectable {{ComputeTaskSession}} in Ignite.NET exposing the {{SetAttributes}} operation similar to what the {{ComputeTaskSession}} does in Java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17657) Partition data loss after rolling restart
Alexey Kukushkin created IGNITE-17657: - Summary: Partition data loss after rolling restart Key: IGNITE-17657 URL: https://issues.apache.org/jira/browse/IGNITE-17657 Project: Ignite Issue Type: Bug Affects Versions: 2.13 Reporter: Alexey Kukushkin *Setup* An active 3+ node Apache Ignite 2.13 cluster with a cache with 1 backup, enabled persistence and default partition loss policy. *Actions* # An application is continuously writing data to the cache # The nodes are sequentially restarted one after another while the data is being written to the cache. The next node is restarted only after the data rebalancing is complete. Using the {{KeysToRebalanceLeft}} metric to monitor rebalancing (see the [documentation|https://ignite.apache.org/docs/latest/monitoring-metrics/metrics#monitoring-rebalancing] for more details) # The application reads some of the data after restarting all the nodes. *Expected* No data is lost since there is 1 backup and the nodes are restarted sequentially after rebalancing is complete. *Actual* Sometimes (in our case in more than 50% of cases) there is a "partition data has been lost" exception on the attempt to read the data. *Notes* Tried to create a JUnit reproducer (all nodes within the same JVM) for the above scenario - no success so far. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17276) CompletableFuture#supplyAsync never completes with Ignite ExecutorService
Alexey Kukushkin created IGNITE-17276: - Summary: CompletableFuture#supplyAsync never completes with Ignite ExecutorService Key: IGNITE-17276 URL: https://issues.apache.org/jira/browse/IGNITE-17276 Project: Ignite Issue Type: Bug Components: compute Affects Versions: 2.13 Reporter: Alexey Kukushkin h3. Scenario: CompletableFuture#supplyAsync with Ignite ExecutorService Given a cluster of two Ignite nodes And Ignite {{ExecutorService}} for a remote node When a task is executed using CompletableFuture#supplyAsync with the Ignite's {{ExecutorService}} Then the task eventually completes Actual result: the task never completes. Waiting on the task's future blocks forever. Note; the scenario works fine if the task is executed on the local Ignite node. {code:java} public class Reproducer { @Test public void supplyIgniteExecutorToCompletableFuture() throws InterruptedException, ExecutionException { Function igniteCfgFactory = igniteName -> new IgniteConfiguration() .setIgniteInstanceName(igniteName) .setDiscoverySpi( new TcpDiscoverySpi() .setIpFinder(new TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))) ); try (var ignored = Ignition.start(igniteCfgFactory.apply("server")); final var ignite = Ignition.start(igniteCfgFactory.apply("app"))) { final var igniteExecutor = ignite.executorService(ignite.cluster().forRemotes()); final var future = CompletableFuture.supplyAsync(new Task(), igniteExecutor); final var result = future.get(); assertNotNull(result); } } private static class Task implements Supplier { @Override public String get() { return UUID.randomUUID().toString(); } } } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17189) Java 17: Untrusted Ignite node never exits after failure to connect to the cluster
Alexey Kukushkin created IGNITE-17189: - Summary: Java 17: Untrusted Ignite node never exits after failure to connect to the cluster Key: IGNITE-17189 URL: https://issues.apache.org/jira/browse/IGNITE-17189 Project: Ignite Issue Type: Bug Components: security Affects Versions: 2.13 Reporter: Alexey Kukushkin *Steps to reproduce* * Start an Ignite server node with SSL enabled * Start another Ignite node with SSL enabled and SSL keys signed by wrong CA authority *Expected* *Actual* *Reproducer* {code:java} @Test public void untrustedServerNeverExitsOnFailureToConnectToCluster() { final Function igniteConfigurationFactory = name -> new IgniteConfiguration() .setIgniteInstanceName(name) .setDiscoverySpi( new TcpDiscoverySpi() .setIpFinder( new TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500..47501")) ) ); final var serverSslContextFactory = new SslContextFactory(); serverSslContextFactory.setProtocols("TLSv1.2"); serverSslContextFactory.setKeyStoreFilePath("server-keys.jks"); serverSslContextFactory.setKeyStorePassword("123456".toCharArray()); serverSslContextFactory.setTrustStoreFilePath("server-trust.jks"); serverSslContextFactory.setTrustStorePassword("123456".toCharArray()); final var tmpSslContextFactory = new SslContextFactory(); tmpSslContextFactory.setProtocols("TLSv1.2"); tmpSslContextFactory.setKeyStoreFilePath("tmp-keys.jks"); tmpSslContextFactory.setKeyStorePassword("123456".toCharArray()); tmpSslContextFactory.setTrustStoreFilePath("tmp-trust.jks"); tmpSslContextFactory.setTrustStorePassword("123456".toCharArray()); final var igniteConfiguration = igniteConfigurationFactory.apply("server1") .setSslContextFactory(tmpSslContextFactory); final var untrustedIgniteConfiguration = igniteConfigurationFactory.apply("server2") .setSslContextFactory(tmpSslContextFactory); try (final var unused = Ignition.start(igniteConfiguration)) { try (final var unused2 = Ignition.start(untrustedIgniteConfiguration)) { } } } {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin resolved IGNITE-15241. --- Resolution: Won't Fix > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary real impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > support the {{CREATE ALIAS}} statement > * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] > This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 > version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and > up to 2.0.202. > * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] > This vulnerability is not applicable to H2 in Ignite since Ignite runs H2 > in embedded mode. H2 cannot be externally exposed in embedded mode. The > vulnerability could be exploited on the local machine where Ignite is > running. However, this limits the severity a lot. > * [CVE-2021-42392|https://www.cvedetails.com/cve/CVE-2021-42392/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > use and does not expose the {{org.h2.util.JdbcUtils.getConnection}} method. > > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, our security policies are very formal and require somehow > addressing the security vulnerabilities anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Description: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{{}ignite-indexing{}}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary real impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] This vulnerability is not applicable to H2 in Ignite since Ignite runs H2 in embedded mode. H2 cannot be externally exposed in embedded mode. The vulnerability could be exploited on the local machine where Ignite is running. However, this limits the severity a lot. * [CVE-2021-42392|https://www.cvedetails.com/cve/CVE-2021-42392/] This vulnerability is not applicable to H2 in Ignite since Ignite does not use and does not expose the {{org.h2.util.JdbcUtils.getConnection}} method. We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. was: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary real impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] This vulnerability is not applicable to H2 in Ignite since Ignite runs H2 in embedded mode. H2 cannot be externally exposed in embedded mode. The vulnerability could be exploited on the local machine where Ignite is running. However, this limits the severity a lot. * [CVE-2021-42392|https://www.cvedetails.com/cve/CVE-2021-42392/] This vulnerability is not applicable to H2 in Ignite since Ignite does not use and does not expose the {{org.h2.util.JdbcUtils.getConnection}} method. We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{{}ignite-indexing{}}}) depends on H2 database > version 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvede
[jira] [Closed] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin closed IGNITE-15241. - Ignite Flags: (was: Docs Required,Release Notes Required) > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary real impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > support the {{CREATE ALIAS}} statement > * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] > This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 > version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and > up to 2.0.202. > * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] > This vulnerability is not applicable to H2 in Ignite since Ignite runs H2 > in embedded mode. H2 cannot be externally exposed in embedded mode. The > vulnerability could be exploited on the local machine where Ignite is > running. However, this limits the severity a lot. > * [CVE-2021-42392|https://www.cvedetails.com/cve/CVE-2021-42392/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > use and does not expose the {{org.h2.util.JdbcUtils.getConnection}} method. > > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, our security policies are very formal and require somehow > addressing the security vulnerabilities anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17542992#comment-17542992 ] Alexey Kukushkin commented on IGNITE-15241: --- The [H2's PR 2227|https://github.com/h2database/h2database/pull/2227] really makes it impossible to upgrade to a newer H2 version where all the vulnerabilities are addressed. I see the following options to address the problem: # None of the vulnerabilities is really applicable to H2 in Apache Ignite due to specifics of how Ignite uses H2. See the impact analysis in the description of this JIRA. This can be used as a justification for appropriate team (DevOps, security) in the organization to add the H2 modules used by Ignite to the list of exceptions of the security vulnerabilities scanner. # H2 module shading: rename the H2 module group or name and replace the default H2 with the new one. This could be done manually or using the [Apache Maven Shade Plugin|https://maven.apache.org/plugins/maven-shade-plugin/]. There is not guarantee that specific security vulnerabilities scanner will not detect such a trick but most likely it will not and the scan would be clean. # [Calcite-based SQL Engine|https://ignite.apache.org/docs/latest/SQL/sql-calcite] was added in Ignite 2.13. This is an alternative to H2 and H2 could be excluded if the Calcite-based engine is configured. The Calcite engine is in beta in release 2.13 and the community wants to announce it as production ready in release 2.14 or 2.15. However, there is no guarantee about that and the release dates are not known at the moment of writing this comment. However, some application development teams may consider trying the Calcite engine and it may prove to be stable enough for them, allowing to get rid of the H2 dependency. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary real impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > support the {{CREATE ALIAS}} statement > * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] > This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 > version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and > up to 2.0.202. > * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] > This vulnerability is not applicable to H2 in Ignite since Ignite runs H2 > in embedded mode. H2 cannot be externally exposed in embedded mode. The > vulnerability could be exploited on the local machine where Ignite is > running. However, this limits the severity a lot. > * [CVE-2021-42392|https://www.cvedetails.com/cve/CVE-2021-42392/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > use and does not expose the {{org.h2.util.JdbcUtils.getConnection}} method. > > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, our security policies are very formal and require somehow > addressing the security vulnerabilities anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Description: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary real impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] This vulnerability is not applicable to H2 in Ignite since Ignite runs H2 in embedded mode. H2 cannot be externally exposed in embedded mode. The vulnerability could be exploited on the local machine where Ignite is running. However, this limits the severity a lot. * [CVE-2021-42392|https://www.cvedetails.com/cve/CVE-2021-42392/] This vulnerability is not applicable to H2 in Ignite since Ignite does not use and does not expose the {{org.h2.util.JdbcUtils.getConnection}} method. We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. was: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary real impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary real impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cv
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Description: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary real impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. was: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary *real* impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary real impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > support the {{CREATE ALIAS}} statement > * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] > This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 > version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and > up to 2.0.202. > * [CVE-2022-23221|https://www.cvedetails.com/cve/CVE-2022-23221/] > > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, ou
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Description: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary *real* impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and up to 2.0.202. * We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. was: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary *real* impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary *real* impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > support the {{CREATE ALIAS}} statement > * [CVE-2021-23463|https://www.cvedetails.com/cve/CVE-2021-23463/] > This vulnerability is not applicable to H2 in Ignite since Ignite uses H2 > version 1.4.197 and the vulnerability is applicable to H2 version 1.4.198 and > up to 2.0.202. > * > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, our security policies are very formal and require somehow > addressing the security vulnerabilities anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Description: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2: !Ignite-H2-Vulnerabilities.png! We did preliminary *real* impact analysis considering how Ignite uses H2: * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] This vulnerability is not applicable to H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] This vulnerability is not applicable to H2 in Ignite since Ignite does not support the {{CREATE ALIAS}} statement We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. was: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2 (see !Ignite-H2-Vulnerabilities.png! : * *Critical* [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] *Impact Analysis*; This vulnerability is not applicable to the H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * d !Ignite-H2-Vulnerabilities.png! We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2: > !Ignite-H2-Vulnerabilities.png! > We did preliminary *real* impact analysis considering how Ignite uses H2: > * [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > store data in H2 and thus there can be no H2 backups in Ignite. > * [CVE-2018-10054|https://www.cvedetails.com/cve/CVE-2018-10054/] > This vulnerability is not applicable to H2 in Ignite since Ignite does not > support the {{CREATE ALIAS}} statement > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, our security policies are very formal and require somehow > addressing the security vulnerabilities anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Description: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197. Black Duck SCA detects these [security vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] in H2 (see !Ignite-H2-Vulnerabilities.png! : * *Critical* [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] *Impact Analysis*; This vulnerability is not applicable to the H2 in Ignite since Ignite does not store data in H2 and thus there can be no H2 backups in Ignite. * d !Ignite-H2-Vulnerabilities.png! We realize all those vulnerabilities are not applicable to H2 in Apache Ignite. However, our security policies are very formal and require somehow addressing the security vulnerabilities anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. was: Upgrade H2 dependency of the ignite-indexing module to the latest version 1.4.200. Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version 1.4.197, which has these two [security vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded as a critical vulnerability by our analyzer (Black Duck SCA) and makes it impossible to use Ignite SQL due to security policies. We realize this vulnerability is probably not even applicable to the H2 in Ignite since there is no H2 database or H2 backups in Ignite. Still the security policies are very formal and do not allow that anyway. We believe there are lots of other enterprises having the same issue. For example, there is another issue IGNITE-14381 referencing the same problem. The latest H2 1.4.200 has no vulnerabilities. > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197. Black Duck SCA detects these [security > vulnerabilities|https://www.cvedetails.com/product/45580/H2database-H2.html?vendor_id=17893] > in H2 (see !Ignite-H2-Vulnerabilities.png! : > * *Critical* [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] > *Impact Analysis*; This vulnerability is not applicable to the H2 in Ignite > since Ignite does not store data in H2 and thus there can be no H2 backups in > Ignite. > * d !Ignite-H2-Vulnerabilities.png! > We realize all those vulnerabilities are not applicable to H2 in Apache > Ignite. However, our security policies are very formal and require somehow > addressing the security vulnerabilities anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Attachment: Ignite-H2-Vulnerabilities.png > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: Ignite-H2-Vulnerabilities.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197, which has these two [security > vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] > [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded > as a critical vulnerability by our analyzer (Black Duck SCA) and makes it > impossible to use Ignite SQL due to security policies. We realize this > vulnerability is probably not even applicable to the H2 in Ignite since there > is no H2 database or H2 backups in Ignite. Still the security policies are > very formal and do not allow that anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Affects Version/s: 2.13 (was: 2.10) > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197, which has these two [security > vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] > [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded > as a critical vulnerability by our analyzer (Black Duck SCA) and makes it > impossible to use Ignite SQL due to security policies. We realize this > vulnerability is probably not even applicable to the H2 in Ignite since there > is no H2 database or H2 backups in Ignite. Still the security policies are > very formal and do not allow that anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin reopened IGNITE-15241: --- I guess the issue was closed by mistake. I do not see how the H2 pull request mentioned in the previous comment addresses this problem > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197, which has these two [security > vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] > [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded > as a critical vulnerability by our analyzer (Black Duck SCA) and makes it > impossible to use Ignite SQL due to security policies. We realize this > vulnerability is probably not even applicable to the H2 in Ignite since there > is no H2 database or H2 backups in Ignite. Still the security policies are > very formal and do not allow that anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16272) IgniteOutOfMemoryException is not a critical failure
[ https://issues.apache.org/jira/browse/IGNITE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16272: -- Remaining Estimate: 48h (was: 40h) Original Estimate: 48h (was: 40h) > IgniteOutOfMemoryException is not a critical failure > > > Key: IGNITE-16272 > URL: https://issues.apache.org/jira/browse/IGNITE-16272 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 48h > Remaining Estimate: 48h > > h3. The Problem > Ignite raises the > [IgniteOutOfMemoryException|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/internal/mem/IgniteOutOfMemoryException.java] > if a data region size is exceeded when trying to add more data to a cache. > Ignite considers the {{IgniteOutOfMemoryException}} as a critical failure. > This causes shutting down the Ignite server with the default failure handler. > However, reaching the data region capacity does not seem to be such a > critical problem requiring the server shutdown or restart. For example, in > our application we just want to report this problem back to the users and > notify the DevOps without applying the critical failure handler. To achieve > that, we had to define a custom {{FailureHandler}} that detects and ignores > the {{IgniteOutOfMemoryException}} and all the exception caused by the > {{{}IgniteOutOfMemoryException{}}}, allowing the final exception to reach the > application. This solution is clumsy and unreliable since it uses the > internal {{IgniteOutOfMemoryException}} definition and relies on a complex > secondary exception structure trying to find the > {{IgniteOutOfMemoryException}} among the suppressed exception and causes. > Ignite out-of-the-box failure handlers have the {{ignoredFailure}} property > that allows filtering out some kinds of failures. However, the > {{IgniteOutOfMemoryException}} is not among the > [FailureType|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/failure/FailureType.java] > that can be ignored. > h3. The Proposal > # Does anyone really want to treat the "data region capacity exceeded" > problem as a critical failure and stop or restart the server? > ** Consider never treating this condition as a critical failure. This change > is not backward compatible. > ** Or add another item to the {{FailureType}} enumeration to optionally > allow the users not to have that treated as a critical failure. This is > backward-compatible. > # Make the {{IgniteOutOfMemoryException}} a public API (now it is in the > {{internal}} package) > # Consider renaming {{IgniteOutOfMemoryException}} (for example, to > something like {{{}NotEnoughStorageException{}}}) since the current name is > similar to a really critical and usually unrecoverable Java's > {{OutOfMemoryError}} although the {{IgniteOutOfMemoryException}} is not that > critical. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15336) TLSv1.3 and Java 11
[ https://issues.apache.org/jira/browse/IGNITE-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15336: -- Remaining Estimate: 64h (was: 48h) Original Estimate: 64h (was: 48h) > TLSv1.3 and Java 11 > --- > > Key: IGNITE-15336 > URL: https://issues.apache.org/jira/browse/IGNITE-15336 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 64h > Remaining Estimate: 64h > > Enhance Ignite to support TLSv1.3 with Java 11. > Ignite currently does not support TLSv1.3 with Java 11. This is > [documented|https://ignite.apache.org/docs/latest/setup#running-ignite-with-java-11-or-later]: > > _TLSv1.3, which is available in Java 11, is not supported at the moment. > Consider adding {{‑Djdk.tls.client.protocols=TLSv1.2}} if SSL between nodes > is used._ > h3. Reproducer > The problem can be reproduced by trying to connect an Ignite Java thin client > to Ignite server over an secure connection. > Add TLSv1.3 to the list of protocols in {{SSLContext}} and set the > {{SSLContext}} in the {{ClientConnectorConfiguration}} on the server and > {{ClientConfiguration}} on the client. > An attempt to {{Ignition.startClient({sslContext: {protocols: > ['TLSv1.3']}})}} fails with a {{ClientConnectionException}} saying _Channel > is closed_. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-14326) Resetting Ignite cache entry TTL
[ https://issues.apache.org/jira/browse/IGNITE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-14326: -- Remaining Estimate: 64h (was: 56h) Original Estimate: 64h (was: 56h) > Resetting Ignite cache entry TTL > > > Key: IGNITE-14326 > URL: https://issues.apache.org/jira/browse/IGNITE-14326 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Stephen Darlington >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Original Estimate: 64h > Remaining Estimate: 64h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to refresh the TTL without first retrieving the record, which > is slow and resource consuming if an entry is large. > Provide a method to reset Ignite cache entry TTL. > Suggested API (to be discussed with community): > {{IgniteCache#touch(key)}}: resets the TTL using the latest TTL value or does > nothing if no TTL was specified for the key. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16431) Entry expiration requires twice the entry size of heap
[ https://issues.apache.org/jira/browse/IGNITE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16431: -- Remaining Estimate: 64h (was: 56h) Original Estimate: 64h (was: 56h) > Entry expiration requires twice the entry size of heap > -- > > Key: IGNITE-16431 > URL: https://issues.apache.org/jira/browse/IGNITE-16431 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: 500MB-put-expiry-master.png > > Original Estimate: 64h > Remaining Estimate: 64h > > Ignite takes twice the entry size off the heap to expire the entry when > {{{}eagerTtl=true{}}}. See the attached heap memory usage diagram of putting > and then expiring a 500MB entry in Ignite. > This makes Ignite inefficient with handling large objects causing > {{OutOfMemory}} errors. > Do we really need loading entry's value on heap at all to expiry the entry? > Please enhance Ignite cache entry expiration not to load the entry's value on > heap even once or explain why it is not possible. > !500MB-put-expiry-master.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16448) Optimize unacknowledged messages queue for FULL_SYNC caches
[ https://issues.apache.org/jira/browse/IGNITE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16448: -- Remaining Estimate: 80h (was: 56h) Original Estimate: 80h (was: 56h) > Optimize unacknowledged messages queue for FULL_SYNC caches > --- > > Key: IGNITE-16448 > URL: https://issues.apache.org/jira/browse/IGNITE-16448 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: apache-ignite-msgReqs-heap-usage.png > > Original Estimate: 80h > Remaining Estimate: 80h > > Apache Ignite queues all unacknowledged messages in the > {{GridNioRecoveryDescriptor#msgReqs}} queue. Size of this queue is > effectively controlled by the {{TcpCommunicationSpi.ackSendThreshold}} > property, which is 32 by default. > Queuing PUTs of large entries results in significant heap memory usage. > Please check if we really need to queue PUTs into FULL_SYNC caches. It seems > to me we should not since a failure to write to a backup would be sent back > to the client right away without any retries. If that is the case then please > optimize the heap usage by not queueing FULL_SYNC cache update messages. > See the heap usage example attached: !apache-ignite-msgReqs-heap-usage.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-14326) Resetting Ignite cache entry TTL
[ https://issues.apache.org/jira/browse/IGNITE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-14326: -- Remaining Estimate: 56h (was: 40h) Original Estimate: 56h (was: 40h) > Resetting Ignite cache entry TTL > > > Key: IGNITE-14326 > URL: https://issues.apache.org/jira/browse/IGNITE-14326 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Stephen Darlington >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Original Estimate: 56h > Remaining Estimate: 56h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to refresh the TTL without first retrieving the record, which > is slow and resource consuming if an entry is large. > Provide a method to reset Ignite cache entry TTL. > Suggested API (to be discussed with community): > {{IgniteCache#touch(key)}}: resets the TTL using the latest TTL value or does > nothing if no TTL was specified for the key. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16137) ContainsKey operation fetches entry value into heap
[ https://issues.apache.org/jira/browse/IGNITE-16137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16137: -- Remaining Estimate: 64h (was: 40h) Original Estimate: 64h (was: 40h) > ContainsKey operation fetches entry value into heap > --- > > Key: IGNITE-16137 > URL: https://issues.apache.org/jira/browse/IGNITE-16137 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 64h > Remaining Estimate: 64h > > [See similar problem for the remove() > operation|https://issues.apache.org/jira/browse/IGNITE-15959] > {{IgniteCache#containsKey(key)}} operation fetches full entry into heap > memory. This is inefficient when working with large objects: our application > running with limited heap memory fails with {{java.lang.OutOfMemoryError: > Java heap space}} when trying to check if a key exists. > It seems wrong that Ignite needs to fetch the full entry on heap to check if > the key exists. Please enhance Ignite to not be doing that or explain why > Ignite must do that. > h2. Reproducer > h3. Steps > Create a Gradle project with the below class and run it as > {{./gradlew test --tests apache.ignite.issues.ContainsOperationHeapUsage}} > {{build.gradle}} > {code:groovy} > test { > minHeapSize = "512m" > maxHeapSize = "512m" > } > {code} > {{ContainsOperationHeapUsage.java}} > {code:java} > public class ContainsOperationHeapUsage { > /** Run with -Xmx512m -Xms512m */ > @Test > public void containsOperationFetchesValueOnHeap() { > var igniteCfg = new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder(new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))) > ) > .setCacheConfiguration(new CacheConfiguration<>("blobs")); > try (var ignite = Ignition.start(igniteCfg)) { > Cache cache = ignite.cache("blobs"); > // Put a BLOB having size of 35% of free memory to the cache > Runtime.getRuntime().gc(); > var freeMemory = Runtime.getRuntime().freeMemory(); > var blobSize = (int)(freeMemory * 0.35); > putBlob(cache, blobSize); > // Use 70% of the free heap > Runtime.getRuntime().gc(); > var unused = new byte[2 * blobSize]; > // Check if the blob exists in the cache. > // This throws "OutOfMemoryError: Java heap space" since Ignite > retrieves full entry to the heap. > // Why does Ignite retrieve entry value to check if the key > exists? > cache.containsKey(1); > } > } > private static void putBlob(Cache cache, int blobSize) { > var blob = new byte[blobSize]; > cache.put(1, blob); > } > } > {code} > h3. Expected > The test passes > h3. Actual > The {{cache.containsKey}} operatoin fails with: > {noformat} > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1385) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:680) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:500) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:411) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:316) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:129) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:422) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetOne.found(BPlusTree.java:3156) > at > org.apache.ignite.internal.processors.cache.persist
[jira] [Updated] (IGNITE-16135) Expiry policy is not applied to Ignite cache after serialization
[ https://issues.apache.org/jira/browse/IGNITE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16135: -- Remaining Estimate: 56h (was: 40h) Original Estimate: 56h (was: 40h) > Expiry policy is not applied to Ignite cache after serialization > > > Key: IGNITE-16135 > URL: https://issues.apache.org/jira/browse/IGNITE-16135 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 56h > Remaining Estimate: 56h > > Expiry Policy specified in Ignite cache operation context is not applied to > the cache after the cache serialization/deserialization. > h3. Steps to Reproduce > # Run an Ignite server with a cache without expiry policy > # A client node joins the server, gets the cache proxy and applies expiry > policy to the cache proxy > # The client executes a Compute task taking the cache proxy as a parameter > on the server. > # The Compute task puts an entry to the cache using the cache proxy received > from the client. > # The client sleeps for a time period longer than the TTL specified for the > expiry policy > # The client checks if the entry that the Compute task added still exists > h3. Expected > The entry does not exist since it must expire > h3. Actual > The entry still exists in the cache. > h3. Reproducer > {code:java} > public class IgniteCacheSerializationTest { > /** Expiry Policy specified in Ignite cache operation context is enabled > after the cache serialization. */ > @Test > public void igniteCacheWithExpiryPolicySerialization() throws > InterruptedException { > var cacheName = "cache1"; > var ttl = 1000; > Supplier igniteCfgFactory = () -> > new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder( > new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500")) > ) > ) > .setCacheConfiguration(new CacheConfiguration<>(cacheName)); > try (var ignored = > Ignition.start(igniteCfgFactory.get().setIgniteInstanceName("server"))) { > try (var ignite = Ignition.start( > > igniteCfgFactory.get().setIgniteInstanceName("test").setClientMode(true)) > ) { > // Apply a 1-second expiry policy to the cache > var expiryPolicy = TouchedExpiryPolicy.factoryOf(new > Duration(TimeUnit.MILLISECONDS, ttl)).create(); > var cache = ignite. String>cache(cacheName).withExpiryPolicy(expiryPolicy); > // Run a Compute Task that serializes the cache with the > 1-second expiry policy applied. > // The task adds a key 1 to the cache > ignite.compute().affinityRun(cacheName, 1, new Put1(cache)); > // Sleep for a time twice longer than the TTL > Thread.sleep(ttl * 2); > // The key 1 must expiry by this time > assertFalse(cache.containsKey(1), "The entry still exists in > the cache"); > } > } > } > private static final class Put1 implements IgniteRunnable { > private final IgniteCache cache; > public Put1(IgniteCache cache) { > this.cache = cache; > } > @Override public void run() { > cache.put(1, "ignored"); > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16715) Ignite should always use IgniteLogger
[ https://issues.apache.org/jira/browse/IGNITE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16715: -- Remaining Estimate: 56h (was: 40h) Original Estimate: 56h (was: 40h) > Ignite should always use IgniteLogger > - > > Key: IGNITE-16715 > URL: https://issues.apache.org/jira/browse/IGNITE-16715 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 56h > Remaining Estimate: 56h > > Ignite sometimes logs directly to stdout (e.g. some startup output). This > makes it hard to setup custom logging. > Please make Ignite always use {{IgniteLogger}} or explain why it cannot be > done. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16716) Ignite JUL Logger should not change the configuration
[ https://issues.apache.org/jira/browse/IGNITE-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16716: -- Remaining Estimate: 32h (was: 24h) Original Estimate: 32h (was: 24h) > Ignite JUL Logger should not change the configuration > - > > Key: IGNITE-16716 > URL: https://issues.apache.org/jira/browse/IGNITE-16716 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 32h > Remaining Estimate: 32h > > Ignite's {{JavaLogger}} forcefully re-configures logging by adding/removing > appenders and changing severity based on some system properties and > environment variables. > This makes it hard or even impossible to consistently switch between > different loggers keeping the same level of output. > Please enhance the {{JavaLogger}} to no mess with the configuration or > explain why that cannot be done -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16922) Getting an entry with expiry policy causes IgniteOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16922: -- Remaining Estimate: 64h (was: 40h) Original Estimate: 64h (was: 40h) > Getting an entry with expiry policy causes IgniteOutOfMemoryException > - > > Key: IGNITE-16922 > URL: https://issues.apache.org/jira/browse/IGNITE-16922 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 64h > Remaining Estimate: 64h > > {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if > {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the > {{key}} and Ignite has not enough storage for another entry of the same or > bigger size. > This happens because: > # Ignite needs to update TTL > # TTL is part of the entry and Ignite overwrites full entry to update the TTL > # The problem is Ignite runs common code that checks if Ignite has enough > storage to write the entry with updated TTL back. The check fails causing the > {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}. > # This behavior is very confusing for Ignite users: why would a "read" > operation throw Ignite OOM? > Can we update the TTL atomically and skip the storage size check? > Please enhance Ignite not to throw Ignite OOM on {{get}}. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16431) Entry expiration requires twice the entry size of heap
[ https://issues.apache.org/jira/browse/IGNITE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16431: -- Remaining Estimate: 56h (was: 40h) Original Estimate: 56h (was: 40h) > Entry expiration requires twice the entry size of heap > -- > > Key: IGNITE-16431 > URL: https://issues.apache.org/jira/browse/IGNITE-16431 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: 500MB-put-expiry-master.png > > Original Estimate: 56h > Remaining Estimate: 56h > > Ignite takes twice the entry size off the heap to expire the entry when > {{{}eagerTtl=true{}}}. See the attached heap memory usage diagram of putting > and then expiring a 500MB entry in Ignite. > This makes Ignite inefficient with handling large objects causing > {{OutOfMemory}} errors. > Do we really need loading entry's value on heap at all to expiry the entry? > Please enhance Ignite cache entry expiration not to load the entry's value on > heap even once or explain why it is not possible. > !500MB-put-expiry-master.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16448) Optimize unacknowledged messages queue for FULL_SYNC caches
[ https://issues.apache.org/jira/browse/IGNITE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16448: -- Remaining Estimate: 56h (was: 40h) Original Estimate: 56h (was: 40h) > Optimize unacknowledged messages queue for FULL_SYNC caches > --- > > Key: IGNITE-16448 > URL: https://issues.apache.org/jira/browse/IGNITE-16448 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: apache-ignite-msgReqs-heap-usage.png > > Original Estimate: 56h > Remaining Estimate: 56h > > Apache Ignite queues all unacknowledged messages in the > {{GridNioRecoveryDescriptor#msgReqs}} queue. Size of this queue is > effectively controlled by the {{TcpCommunicationSpi.ackSendThreshold}} > property, which is 32 by default. > Queuing PUTs of large entries results in significant heap memory usage. > Please check if we really need to queue PUTs into FULL_SYNC caches. It seems > to me we should not since a failure to write to a backup would be sent back > to the client right away without any retries. If that is the case then please > optimize the heap usage by not queueing FULL_SYNC cache update messages. > See the heap usage example attached: !apache-ignite-msgReqs-heap-usage.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Remaining Estimate: 80h (was: 336h) Original Estimate: 80h (was: 336h) > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 80h > Remaining Estimate: 80h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197, which has these two [security > vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] > [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded > as a critical vulnerability by our analyzer (Black Duck SCA) and makes it > impossible to use Ignite SQL due to security policies. We realize this > vulnerability is probably not even applicable to the H2 in Ignite since there > is no H2 database or H2 backups in Ignite. Still the security policies are > very formal and do not allow that anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16137) ContainsKey operation fetches entry value into heap
[ https://issues.apache.org/jira/browse/IGNITE-16137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16137: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > ContainsKey operation fetches entry value into heap > --- > > Key: IGNITE-16137 > URL: https://issues.apache.org/jira/browse/IGNITE-16137 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > [See similar problem for the remove() > operation|https://issues.apache.org/jira/browse/IGNITE-15959] > {{IgniteCache#containsKey(key)}} operation fetches full entry into heap > memory. This is inefficient when working with large objects: our application > running with limited heap memory fails with {{java.lang.OutOfMemoryError: > Java heap space}} when trying to check if a key exists. > It seems wrong that Ignite needs to fetch the full entry on heap to check if > the key exists. Please enhance Ignite to not be doing that or explain why > Ignite must do that. > h2. Reproducer > h3. Steps > Create a Gradle project with the below class and run it as > {{./gradlew test --tests apache.ignite.issues.ContainsOperationHeapUsage}} > {{build.gradle}} > {code:groovy} > test { > minHeapSize = "512m" > maxHeapSize = "512m" > } > {code} > {{ContainsOperationHeapUsage.java}} > {code:java} > public class ContainsOperationHeapUsage { > /** Run with -Xmx512m -Xms512m */ > @Test > public void containsOperationFetchesValueOnHeap() { > var igniteCfg = new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder(new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))) > ) > .setCacheConfiguration(new CacheConfiguration<>("blobs")); > try (var ignite = Ignition.start(igniteCfg)) { > Cache cache = ignite.cache("blobs"); > // Put a BLOB having size of 35% of free memory to the cache > Runtime.getRuntime().gc(); > var freeMemory = Runtime.getRuntime().freeMemory(); > var blobSize = (int)(freeMemory * 0.35); > putBlob(cache, blobSize); > // Use 70% of the free heap > Runtime.getRuntime().gc(); > var unused = new byte[2 * blobSize]; > // Check if the blob exists in the cache. > // This throws "OutOfMemoryError: Java heap space" since Ignite > retrieves full entry to the heap. > // Why does Ignite retrieve entry value to check if the key > exists? > cache.containsKey(1); > } > } > private static void putBlob(Cache cache, int blobSize) { > var blob = new byte[blobSize]; > cache.put(1, blob); > } > } > {code} > h3. Expected > The test passes > h3. Actual > The {{cache.containsKey}} operatoin fails with: > {noformat} > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1385) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:680) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:500) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:411) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:316) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:129) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:422) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetOne.found(BPlusTree.java:3156) > at > org.apache.ignite.internal.processors.cache.persi
[jira] [Updated] (IGNITE-15959) Remove operation fetches entry value into heap
[ https://issues.apache.org/jira/browse/IGNITE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15959: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > Remove operation fetches entry value into heap > -- > > Key: IGNITE-15959 > URL: https://issues.apache.org/jira/browse/IGNITE-15959 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > [See similar problem for the containsKey() > operation|https://issues.apache.org/jira/browse/IGNITE-16137] > {{IgniteCache#remove(key)}} operation fetches full entry into heap memory. > This is inefficient when working with large objects: our application running > with limited heap memory fails with {{java.lang.OutOfMemoryError: Java heap > space}} when trying to remove an entry by key. > It seems wrong that Ignite needs to fetch the full entry on heap to remove > the entry. Please enhance Ignite to not be doing that or explain why Ignite > must do that. > h2. Reproducer > h3. Steps > Create a Gradle project with the below class and run it as > {{./gradlew test --tests apache.ignite.issues.RemoveOperationHeapUsage}} > {{build.gradle}} > {code:groovy} > test { > minHeapSize = "512m" > maxHeapSize = "512m" > } > {code} > {{RemoveOperationHeapUsage.java}} > {code:java} > public class RemoveOperationHeapUsage { > /** Run the test with -Xmx512m -Xms512m */ > @Test > public void removeOperationFetchesValueOnHeap() { > var igniteCfg = new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder(new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))) > ) > .setCacheConfiguration(new CacheConfiguration<>("blobs")); > try (var ignite = Ignition.start(igniteCfg)) { > Cache cache = ignite.cache("blobs"); > // Put a BLOB having size of 35% of free memory to the cache > Runtime.getRuntime().gc(); > var freeMemory = Runtime.getRuntime().freeMemory(); > var blobSize = (int)(freeMemory * 0.35); > putBlob(cache, blobSize); > // Use 70% of the free heap > Runtime.getRuntime().gc(); > var unused = new byte[2 * blobSize]; > // Remove the blob from the cache. > // This throws "OutOfMemoryError: Java heap space" since Ignite > retrieves full entry to the heap. > // Why does Ignite retrieve entry value to delete the entry? > cache.remove(1); > } > } > private static void putBlob(Cache cache, int blobSize) { > var blob = new byte[blobSize]; > cache.put(1, blob); > } > } > {code} > h3. Expected > The test passes > h3. Actual > The {{cache.remove}} operatoin fails with: > {noformat} > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1385) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:680) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:500) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:411) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:316) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:129) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:422) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.found(BPlusTree.java:3987) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:317) > at > org
[jira] [Updated] (IGNITE-16135) Expiry policy is not applied to Ignite cache after serialization
[ https://issues.apache.org/jira/browse/IGNITE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16135: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > Expiry policy is not applied to Ignite cache after serialization > > > Key: IGNITE-16135 > URL: https://issues.apache.org/jira/browse/IGNITE-16135 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > Expiry Policy specified in Ignite cache operation context is not applied to > the cache after the cache serialization/deserialization. > h3. Steps to Reproduce > # Run an Ignite server with a cache without expiry policy > # A client node joins the server, gets the cache proxy and applies expiry > policy to the cache proxy > # The client executes a Compute task taking the cache proxy as a parameter > on the server. > # The Compute task puts an entry to the cache using the cache proxy received > from the client. > # The client sleeps for a time period longer than the TTL specified for the > expiry policy > # The client checks if the entry that the Compute task added still exists > h3. Expected > The entry does not exist since it must expire > h3. Actual > The entry still exists in the cache. > h3. Reproducer > {code:java} > public class IgniteCacheSerializationTest { > /** Expiry Policy specified in Ignite cache operation context is enabled > after the cache serialization. */ > @Test > public void igniteCacheWithExpiryPolicySerialization() throws > InterruptedException { > var cacheName = "cache1"; > var ttl = 1000; > Supplier igniteCfgFactory = () -> > new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder( > new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500")) > ) > ) > .setCacheConfiguration(new CacheConfiguration<>(cacheName)); > try (var ignored = > Ignition.start(igniteCfgFactory.get().setIgniteInstanceName("server"))) { > try (var ignite = Ignition.start( > > igniteCfgFactory.get().setIgniteInstanceName("test").setClientMode(true)) > ) { > // Apply a 1-second expiry policy to the cache > var expiryPolicy = TouchedExpiryPolicy.factoryOf(new > Duration(TimeUnit.MILLISECONDS, ttl)).create(); > var cache = ignite. String>cache(cacheName).withExpiryPolicy(expiryPolicy); > // Run a Compute Task that serializes the cache with the > 1-second expiry policy applied. > // The task adds a key 1 to the cache > ignite.compute().affinityRun(cacheName, 1, new Put1(cache)); > // Sleep for a time twice longer than the TTL > Thread.sleep(ttl * 2); > // The key 1 must expiry by this time > assertFalse(cache.containsKey(1), "The entry still exists in > the cache"); > } > } > } > private static final class Put1 implements IgniteRunnable { > private final IgniteCache cache; > public Put1(IgniteCache cache) { > this.cache = cache; > } > @Override public void run() { > cache.put(1, "ignored"); > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-14326) Resetting Ignite cache entry TTL
[ https://issues.apache.org/jira/browse/IGNITE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-14326: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > Resetting Ignite cache entry TTL > > > Key: IGNITE-14326 > URL: https://issues.apache.org/jira/browse/IGNITE-14326 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Stephen Darlington >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to refresh the TTL without first retrieving the record, which > is slow and resource consuming if an entry is large. > Provide a method to reset Ignite cache entry TTL. > Suggested API (to be discussed with community): > {{IgniteCache#touch(key)}}: resets the TTL using the latest TTL value or does > nothing if no TTL was specified for the key. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > SSLException: closing inbound before receiving peer's close_notify > -- > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. Analysis > The problem _probably_ happens due to [this > code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] > calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} > alert, which TLS 1.2 is expecting (see [RFC > 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]). I guess the > right approach to close an SSL socket is just calling {{Socke#close}}, which > should properly wait/send a {{close_notify}} > Some references to consider: > [JDK-8215102 Closing connection to Mysql database results in > exception|https://bugs.openjdk.java.net/browse/JDK-8215102] > [Fix for #93590 - ignore javax.net.ssl.SSLException: closing inbound before > receiving peer's close_notify on > java11+|https://github.com/mysql/mysql-connector-j/pull/32] > [JDK-8251553 Socket closure issues in migrating from JDK 8 to JDK > 11|https://bugs.openjdk.java.net/browse/JDK-8251553] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16715) Ignite should always use IgniteLogger
[ https://issues.apache.org/jira/browse/IGNITE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16715: -- Remaining Estimate: 40h (was: 72h) Original Estimate: 40h (was: 72h) > Ignite should always use IgniteLogger > - > > Key: IGNITE-16715 > URL: https://issues.apache.org/jira/browse/IGNITE-16715 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > Ignite sometimes logs directly to stdout (e.g. some startup output). This > makes it hard to setup custom logging. > Please make Ignite always use {{IgniteLogger}} or explain why it cannot be > done. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16431) Entry expiration requires twice the entry size of heap
[ https://issues.apache.org/jira/browse/IGNITE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16431: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > Entry expiration requires twice the entry size of heap > -- > > Key: IGNITE-16431 > URL: https://issues.apache.org/jira/browse/IGNITE-16431 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: 500MB-put-expiry-master.png > > Original Estimate: 40h > Remaining Estimate: 40h > > Ignite takes twice the entry size off the heap to expire the entry when > {{{}eagerTtl=true{}}}. See the attached heap memory usage diagram of putting > and then expiring a 500MB entry in Ignite. > This makes Ignite inefficient with handling large objects causing > {{OutOfMemory}} errors. > Do we really need loading entry's value on heap at all to expiry the entry? > Please enhance Ignite cache entry expiration not to load the entry's value on > heap even once or explain why it is not possible. > !500MB-put-expiry-master.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16448) Optimize unacknowledged messages queue for FULL_SYNC caches
[ https://issues.apache.org/jira/browse/IGNITE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16448: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > Optimize unacknowledged messages queue for FULL_SYNC caches > --- > > Key: IGNITE-16448 > URL: https://issues.apache.org/jira/browse/IGNITE-16448 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: apache-ignite-msgReqs-heap-usage.png > > Original Estimate: 40h > Remaining Estimate: 40h > > Apache Ignite queues all unacknowledged messages in the > {{GridNioRecoveryDescriptor#msgReqs}} queue. Size of this queue is > effectively controlled by the {{TcpCommunicationSpi.ackSendThreshold}} > property, which is 32 by default. > Queuing PUTs of large entries results in significant heap memory usage. > Please check if we really need to queue PUTs into FULL_SYNC caches. It seems > to me we should not since a failure to write to a backup would be sent back > to the client right away without any retries. If that is the case then please > optimize the heap usage by not queueing FULL_SYNC cache update messages. > See the heap usage example attached: !apache-ignite-msgReqs-heap-usage.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16272) IgniteOutOfMemoryException is not a critical failure
[ https://issues.apache.org/jira/browse/IGNITE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16272: -- Remaining Estimate: 40h (was: 168h) Original Estimate: 40h (was: 168h) > IgniteOutOfMemoryException is not a critical failure > > > Key: IGNITE-16272 > URL: https://issues.apache.org/jira/browse/IGNITE-16272 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > h3. The Problem > Ignite raises the > [IgniteOutOfMemoryException|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/internal/mem/IgniteOutOfMemoryException.java] > if a data region size is exceeded when trying to add more data to a cache. > Ignite considers the {{IgniteOutOfMemoryException}} as a critical failure. > This causes shutting down the Ignite server with the default failure handler. > However, reaching the data region capacity does not seem to be such a > critical problem requiring the server shutdown or restart. For example, in > our application we just want to report this problem back to the users and > notify the DevOps without applying the critical failure handler. To achieve > that, we had to define a custom {{FailureHandler}} that detects and ignores > the {{IgniteOutOfMemoryException}} and all the exception caused by the > {{{}IgniteOutOfMemoryException{}}}, allowing the final exception to reach the > application. This solution is clumsy and unreliable since it uses the > internal {{IgniteOutOfMemoryException}} definition and relies on a complex > secondary exception structure trying to find the > {{IgniteOutOfMemoryException}} among the suppressed exception and causes. > Ignite out-of-the-box failure handlers have the {{ignoredFailure}} property > that allows filtering out some kinds of failures. However, the > {{IgniteOutOfMemoryException}} is not among the > [FailureType|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/failure/FailureType.java] > that can be ignored. > h3. The Proposal > # Does anyone really want to treat the "data region capacity exceeded" > problem as a critical failure and stop or restart the server? > ** Consider never treating this condition as a critical failure. This change > is not backward compatible. > ** Or add another item to the {{FailureType}} enumeration to optionally > allow the users not to have that treated as a critical failure. This is > backward-compatible. > # Make the {{IgniteOutOfMemoryException}} a public API (now it is in the > {{internal}} package) > # Consider renaming {{IgniteOutOfMemoryException}} (for example, to > something like {{{}NotEnoughStorageException{}}}) since the current name is > similar to a really critical and usually unrecoverable Java's > {{OutOfMemoryError}} although the {{IgniteOutOfMemoryException}} is not that > critical. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16889) Error-prone X#searchForCause() implementation
[ https://issues.apache.org/jira/browse/IGNITE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16889: -- Remaining Estimate: 16h (was: 24h) Original Estimate: 16h (was: 24h) > Error-prone X#searchForCause() implementation > - > > Key: IGNITE-16889 > URL: https://issues.apache.org/jira/browse/IGNITE-16889 > Project: Ignite > Issue Type: Bug > Components: general >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 16h > Remaining Estimate: 16h > > The {{X#seachForCause}} utility method fails with a {{NullPointerException}} > if the analyzed {{Throwable}} has no {{suppressed}} exception due to the > [missing null > check|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/typedef/X.java#L570]. > I never encountered any issues with that with real usage of Ignite. However, > writing a unit test that stabs the {{IgniteCache}} methods to throw an > exception without initialized {{suppressed}} exceptions results in the > stabbed method to return {{null}} instead of throwing an exception. The > workaround is to properly initialize the {{suppressed}} list in the mocked > exception. > I suggest to make the generic {{X#seachForCause}} utility method handle the > case when the {{suppressed}} list is not initialized. That would simplify the > unit tests code. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16922) Getting an entry with expiry policy causes IgniteOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16922: -- Remaining Estimate: 40h (was: 72h) Original Estimate: 40h (was: 72h) > Getting an entry with expiry policy causes IgniteOutOfMemoryException > - > > Key: IGNITE-16922 > URL: https://issues.apache.org/jira/browse/IGNITE-16922 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 40h > Remaining Estimate: 40h > > {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if > {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the > {{key}} and Ignite has not enough storage for another entry of the same or > bigger size. > This happens because: > # Ignite needs to update TTL > # TTL is part of the entry and Ignite overwrites full entry to update the TTL > # The problem is Ignite runs common code that checks if Ignite has enough > storage to write the entry with updated TTL back. The check fails causing the > {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}. > # This behavior is very confusing for Ignite users: why would a "read" > operation throw Ignite OOM? > Can we update the TTL atomically and skip the storage size check? > Please enhance Ignite not to throw Ignite OOM on {{get}}. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16716) Ignite JUL Logger should not change the configuration
[ https://issues.apache.org/jira/browse/IGNITE-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16716: -- Remaining Estimate: 24h (was: 48h) Original Estimate: 24h (was: 48h) > Ignite JUL Logger should not change the configuration > - > > Key: IGNITE-16716 > URL: https://issues.apache.org/jira/browse/IGNITE-16716 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 24h > Remaining Estimate: 24h > > Ignite's {{JavaLogger}} forcefully re-configures logging by adding/removing > appenders and changing severity based on some system properties and > environment variables. > This makes it hard or even impossible to consistently switch between > different loggers keeping the same level of output. > Please enhance the {{JavaLogger}} to no mess with the configuration or > explain why that cannot be done -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15962) CacheEntryRemovedListener makes the PUT operation to create ENTRY CREATED events
[ https://issues.apache.org/jira/browse/IGNITE-15962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15962: -- Remaining Estimate: 48h (was: 72h) Original Estimate: 48h (was: 72h) > CacheEntryRemovedListener makes the PUT operation to create ENTRY CREATED > events > > > Key: IGNITE-15962 > URL: https://issues.apache.org/jira/browse/IGNITE-15962 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: CacheContinuousQueryEntry-HeapDump.png, > CacheContinuousQueryEntry-StackTrace.png, EventListenerHeapUsage.java, > IgniteServer.java, build.gradle > > Original Estimate: 48h > Remaining Estimate: 48h > > {{CacheEntryRemovedListener}} configured with > {{MutableCacheEntryListenerConfiguration.isOldValueRequired = false}} makes > the {{Cache#put}} operation to create {{CREATED}} events on the primary > server for a short period of time. This is inefficient when working with > large objects: our application running with limited heap memory fails with > {{java.lang.OutOfMemoryError: Java heap space}} after executing a {{PUT}} > operation since the entry referencing the object is "stuck" on the heap. > h2. Reproducer > h3. Steps > We need two Ignite server nodes to demonstrate the problem. A cache is > deployed on both the nodes and both the nodes listen to the {{ENTRY REMOVED}} > events. Both the nodes disable fetching the {{CacheEntryEvent#getOldValue}} > by setting {{MutableCacheEntryListenerConfiguration.isOldValueRequired = > false}} > # Build and run the attached [^IgniteServer.java]. This server is configured > like described above and runs until manually terminated. > # Build the run the attached test [^EventListenerHeapUsage.java] . This test: > ## Starts another server configured like described above > ## Puts a large object taking 35% of the free memory. This is done in an > isolated scope so the memory after the operation is garbage collected, > ## Attempts to allocate 70% of the heap. > h3. Expected > The last "allocate 70% of the heap" operation should succeed. > h3. Actual > {noformat} > Caused by: java.lang.OutOfMemoryError: Java heap space > at > apache.ignite.issues.EventListenerHeapUsage.entryRemovedEventListenerFetchesValueOnHeap(EventListenerHeapUsage.java:33) > {noformat} > h2 Analysis > For some reason configuring the event listener makes the {{IgniteCache#put}} > operation to create and attach an instance of {{CacheContinuousQueryEntry}} > with {{evtType=CREATED}} and {{newObject}} referencing the entry. > The attached heap dump !CacheContinuousQueryEntry-HeapDump.png! collected on > the {{OutOfMemoryError}} demonstrates the large object "stuck" on the heap. > The attached {{CacheContinuousQueryEntry}} initialization > !CacheContinuousQueryEntry-StackTrace.png! proves the {{ENTRY CREATED}} event > is being created even though we did not configure it. > It seems wrong that Ignite needs to attached the {{ENTRY CREATED}} event to > the "update" message. Please enhance Ignite to not be doing that or explain > why Ignite must do that. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15336) TLSv1.3 and Java 11
[ https://issues.apache.org/jira/browse/IGNITE-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15336: -- Remaining Estimate: 48h (was: 168h) Original Estimate: 48h (was: 168h) The issues is probably already fixed. The described note was removed from Ignite documentation. Need to confirm and close this without a fix if that is the case. > TLSv1.3 and Java 11 > --- > > Key: IGNITE-15336 > URL: https://issues.apache.org/jira/browse/IGNITE-15336 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 48h > Remaining Estimate: 48h > > Enhance Ignite to support TLSv1.3 with Java 11. > Ignite currently does not support TLSv1.3 with Java 11. This is > [documented|https://ignite.apache.org/docs/latest/setup#running-ignite-with-java-11-or-later]: > > _TLSv1.3, which is available in Java 11, is not supported at the moment. > Consider adding {{‑Djdk.tls.client.protocols=TLSv1.2}} if SSL between nodes > is used._ > h3. Reproducer > The problem can be reproduced by trying to connect an Ignite Java thin client > to Ignite server over an secure connection. > Add TLSv1.3 to the list of protocols in {{SSLContext}} and set the > {{SSLContext}} in the {{ClientConnectorConfiguration}} on the server and > {{ClientConfiguration}} on the client. > An attempt to {{Ignition.startClient({sslContext: {protocols: > ['TLSv1.3']}})}} fails with a {{ClientConnectionException}} saying _Channel > is closed_. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16715) Ignite should always use IgniteLogger
[ https://issues.apache.org/jira/browse/IGNITE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16715: -- Remaining Estimate: 72h (was: 168h) Original Estimate: 72h (was: 168h) > Ignite should always use IgniteLogger > - > > Key: IGNITE-16715 > URL: https://issues.apache.org/jira/browse/IGNITE-16715 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 72h > Remaining Estimate: 72h > > Ignite sometimes logs directly to stdout (e.g. some startup output). This > makes it hard to setup custom logging. > Please make Ignite always use {{IgniteLogger}} or explain why it cannot be > done. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16716) Ignite JUL Logger should not change the configuration
[ https://issues.apache.org/jira/browse/IGNITE-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16716: -- Remaining Estimate: 48h (was: 168h) Original Estimate: 48h (was: 168h) > Ignite JUL Logger should not change the configuration > - > > Key: IGNITE-16716 > URL: https://issues.apache.org/jira/browse/IGNITE-16716 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 48h > Remaining Estimate: 48h > > Ignite's {{JavaLogger}} forcefully re-configures logging by adding/removing > appenders and changing severity based on some system properties and > environment variables. > This makes it hard or even impossible to consistently switch between > different loggers keeping the same level of output. > Please enhance the {{JavaLogger}} to no mess with the configuration or > explain why that cannot be done -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16922) Getting an entry with expiry policy causes IgniteOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16922: -- Remaining Estimate: 72h (was: 168h) Original Estimate: 72h (was: 168h) > Getting an entry with expiry policy causes IgniteOutOfMemoryException > - > > Key: IGNITE-16922 > URL: https://issues.apache.org/jira/browse/IGNITE-16922 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 72h > Remaining Estimate: 72h > > {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if > {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the > {{key}} and Ignite has not enough storage for another entry of the same or > bigger size. > This happens because: > # Ignite needs to update TTL > # TTL is part of the entry and Ignite overwrites full entry to update the TTL > # The problem is Ignite runs common code that checks if Ignite has enough > storage to write the entry with updated TTL back. The check fails causing the > {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}. > # This behavior is very confusing for Ignite users: why would a "read" > operation throw Ignite OOM? > Can we update the TTL atomically and skip the storage size check? > Please enhance Ignite not to throw Ignite OOM on {{get}}. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (IGNITE-7641) Add CacheEntry#ttl method
[ https://issues.apache.org/jira/browse/IGNITE-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin closed IGNITE-7641. > Add CacheEntry#ttl method > - > > Key: IGNITE-7641 > URL: https://issues.apache.org/jira/browse/IGNITE-7641 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Valentin Kulichenko >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Time Spent: 50m > Remaining Estimate: 168h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to know the current TTL for a particular key. > We can add {{CacheEntry#ttl()}} and/or {{IgniteCache#ttl(K key)}} method that > will provide this information. Looks like it's already available via > {{GridCacheMapEntry#ttl()}}, so we just need to properly expose it to public > API. > Here is the user forum discussion about this: > http://apache-ignite-users.70518.x6.nabble.com/Get-TTL-of-the-specific-K-V-entry-td19817.html -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (IGNITE-7641) Add CacheEntry#ttl method
[ https://issues.apache.org/jira/browse/IGNITE-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin resolved IGNITE-7641. -- Resolution: Duplicate The requirement is described in IGNITE-14326. Closing this issue as a duplicate > Add CacheEntry#ttl method > - > > Key: IGNITE-7641 > URL: https://issues.apache.org/jira/browse/IGNITE-7641 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Valentin Kulichenko >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Time Spent: 50m > Remaining Estimate: 168h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to know the current TTL for a particular key. > We can add {{CacheEntry#ttl()}} and/or {{IgniteCache#ttl(K key)}} method that > will provide this information. Looks like it's already available via > {{GridCacheMapEntry#ttl()}}, so we just need to properly expose it to public > API. > Here is the user forum discussion about this: > http://apache-ignite-users.70518.x6.nabble.com/Get-TTL-of-the-specific-K-V-entry-td19817.html -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-14326) Resetting Ignite cache entry TTL
[ https://issues.apache.org/jira/browse/IGNITE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-14326: -- Description: Ignite provides a way to specify an expiry policy on per entry level, but there is no way to refresh the TTL without first retrieving the record, which is slow and resource consuming if an entry is large. Provide a method to reset Ignite cache entry TTL. Suggested API (to be discussed with community): {{IgniteCache#touch(key)}}: resets the TTL using the latest TTL value or does nothing if no TTL was specified for the key. was: Ignite provides a way to specify an expiry policy on per entry level, but there is no way to know the current TTL for a particular key or refresh the TTL without first retrieving the record. Ticket IGNITE-7641 details the IgniteCache#ttl() method. This ticket is about _setting_ the TTL. The API for this is less well defined than the get TTL method. Suggest a number of options: * Update using the default TTL policy * Update using a specified long * Update using a specified Expiry policy > Resetting Ignite cache entry TTL > > > Key: IGNITE-14326 > URL: https://issues.apache.org/jira/browse/IGNITE-14326 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Stephen Darlington >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to refresh the TTL without first retrieving the record, which > is slow and resource consuming if an entry is large. > Provide a method to reset Ignite cache entry TTL. > Suggested API (to be discussed with community): > {{IgniteCache#touch(key)}}: resets the TTL using the latest TTL value or does > nothing if no TTL was specified for the key. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-14326) Resetting Ignite cache entry TTL
[ https://issues.apache.org/jira/browse/IGNITE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-14326: -- Summary: Resetting Ignite cache entry TTL (was: Add CacheEntry#setTtl method) > Resetting Ignite cache entry TTL > > > Key: IGNITE-14326 > URL: https://issues.apache.org/jira/browse/IGNITE-14326 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Stephen Darlington >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to know the current TTL for a particular key or refresh the > TTL without first retrieving the record. > Ticket IGNITE-7641 details the IgniteCache#ttl() method. This ticket is about > _setting_ the TTL. > The API for this is less well defined than the get TTL method. Suggest a > number of options: > * Update using the default TTL policy > * Update using a specified long > * Update using a specified Expiry policy -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15336) TLSv1.3 and Java 11
[ https://issues.apache.org/jira/browse/IGNITE-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15336: -- Remaining Estimate: 168h (was: 336h) Original Estimate: 168h (was: 336h) > TLSv1.3 and Java 11 > --- > > Key: IGNITE-15336 > URL: https://issues.apache.org/jira/browse/IGNITE-15336 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Enhance Ignite to support TLSv1.3 with Java 11. > Ignite currently does not support TLSv1.3 with Java 11. This is > [documented|https://ignite.apache.org/docs/latest/setup#running-ignite-with-java-11-or-later]: > > _TLSv1.3, which is available in Java 11, is not supported at the moment. > Consider adding {{‑Djdk.tls.client.protocols=TLSv1.2}} if SSL between nodes > is used._ > h3. Reproducer > The problem can be reproduced by trying to connect an Ignite Java thin client > to Ignite server over an secure connection. > Add TLSv1.3 to the list of protocols in {{SSLContext}} and set the > {{SSLContext}} in the {{ClientConnectorConfiguration}} on the server and > {{ClientConfiguration}} on the client. > An attempt to {{Ignition.startClient({sslContext: {protocols: > ['TLSv1.3']}})}} fails with a {{ClientConnectionException}} saying _Channel > is closed_. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15336) TLSv1.3 and Java 11
[ https://issues.apache.org/jira/browse/IGNITE-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15336: -- Remaining Estimate: 336h Original Estimate: 336h > TLSv1.3 and Java 11 > --- > > Key: IGNITE-15336 > URL: https://issues.apache.org/jira/browse/IGNITE-15336 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 336h > Remaining Estimate: 336h > > Enhance Ignite to support TLSv1.3 with Java 11. > Ignite currently does not support TLSv1.3 with Java 11. This is > [documented|https://ignite.apache.org/docs/latest/setup#running-ignite-with-java-11-or-later]: > > _TLSv1.3, which is available in Java 11, is not supported at the moment. > Consider adding {{‑Djdk.tls.client.protocols=TLSv1.2}} if SSL between nodes > is used._ > h3. Reproducer > The problem can be reproduced by trying to connect an Ignite Java thin client > to Ignite server over an secure connection. > Add TLSv1.3 to the list of protocols in {{SSLContext}} and set the > {{SSLContext}} in the {{ClientConnectorConfiguration}} on the server and > {{ClientConfiguration}} on the client. > An attempt to {{Ignition.startClient({sslContext: {protocols: > ['TLSv1.3']}})}} fails with a {{ClientConnectionException}} saying _Channel > is closed_. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-14326) Add CacheEntry#setTtl method
[ https://issues.apache.org/jira/browse/IGNITE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-14326: -- Remaining Estimate: 168h Original Estimate: 168h > Add CacheEntry#setTtl method > > > Key: IGNITE-14326 > URL: https://issues.apache.org/jira/browse/IGNITE-14326 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Stephen Darlington >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to know the current TTL for a particular key or refresh the > TTL without first retrieving the record. > Ticket IGNITE-7641 details the IgniteCache#ttl() method. This ticket is about > _setting_ the TTL. > The API for this is less well defined than the get TTL method. Suggest a > number of options: > * Update using the default TTL policy > * Update using a specified long > * Update using a specified Expiry policy -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Remaining Estimate: 336h Original Estimate: 336h > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 336h > Remaining Estimate: 336h > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197, which has these two [security > vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] > [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded > as a critical vulnerability by our analyzer (Black Duck SCA) and makes it > impossible to use Ignite SQL due to security policies. We realize this > vulnerability is probably not even applicable to the H2 in Ignite since there > is no H2 database or H2 backups in Ignite. Still the security policies are > very formal and do not allow that anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-7641) Add CacheEntry#ttl method
[ https://issues.apache.org/jira/browse/IGNITE-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-7641: - Remaining Estimate: 168h (was: 0h) > Add CacheEntry#ttl method > - > > Key: IGNITE-7641 > URL: https://issues.apache.org/jira/browse/IGNITE-7641 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.3 >Reporter: Valentin Kulichenko >Assignee: Stephen Darlington >Priority: Major > Labels: cggg > Time Spent: 50m > Remaining Estimate: 168h > > Ignite provides a way to specify an expiry policy on per entry level, but > there is no way to know the current TTL for a particular key. > We can add {{CacheEntry#ttl()}} and/or {{IgniteCache#ttl(K key)}} method that > will provide this information. Looks like it's already available via > {{GridCacheMapEntry#ttl()}}, so we just need to properly expose it to public > API. > Here is the user forum discussion about this: > http://apache-ignite-users.70518.x6.nabble.com/Get-TTL-of-the-specific-K-V-entry-td19817.html -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-15959) Remove operation fetches entry value into heap
[ https://issues.apache.org/jira/browse/IGNITE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15959: -- Remaining Estimate: 168h Original Estimate: 168h > Remove operation fetches entry value into heap > -- > > Key: IGNITE-15959 > URL: https://issues.apache.org/jira/browse/IGNITE-15959 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > [See similar problem for the containsKey() > operation|https://issues.apache.org/jira/browse/IGNITE-16137] > {{IgniteCache#remove(key)}} operation fetches full entry into heap memory. > This is inefficient when working with large objects: our application running > with limited heap memory fails with {{java.lang.OutOfMemoryError: Java heap > space}} when trying to remove an entry by key. > It seems wrong that Ignite needs to fetch the full entry on heap to remove > the entry. Please enhance Ignite to not be doing that or explain why Ignite > must do that. > h2. Reproducer > h3. Steps > Create a Gradle project with the below class and run it as > {{./gradlew test --tests apache.ignite.issues.RemoveOperationHeapUsage}} > {{build.gradle}} > {code:groovy} > test { > minHeapSize = "512m" > maxHeapSize = "512m" > } > {code} > {{RemoveOperationHeapUsage.java}} > {code:java} > public class RemoveOperationHeapUsage { > /** Run the test with -Xmx512m -Xms512m */ > @Test > public void removeOperationFetchesValueOnHeap() { > var igniteCfg = new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder(new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))) > ) > .setCacheConfiguration(new CacheConfiguration<>("blobs")); > try (var ignite = Ignition.start(igniteCfg)) { > Cache cache = ignite.cache("blobs"); > // Put a BLOB having size of 35% of free memory to the cache > Runtime.getRuntime().gc(); > var freeMemory = Runtime.getRuntime().freeMemory(); > var blobSize = (int)(freeMemory * 0.35); > putBlob(cache, blobSize); > // Use 70% of the free heap > Runtime.getRuntime().gc(); > var unused = new byte[2 * blobSize]; > // Remove the blob from the cache. > // This throws "OutOfMemoryError: Java heap space" since Ignite > retrieves full entry to the heap. > // Why does Ignite retrieve entry value to delete the entry? > cache.remove(1); > } > } > private static void putBlob(Cache cache, int blobSize) { > var blob = new byte[blobSize]; > cache.put(1, blob); > } > } > {code} > h3. Expected > The test passes > h3. Actual > The {{cache.remove}} operatoin fails with: > {noformat} > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1385) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:680) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:500) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:411) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:316) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:129) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:422) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.found(BPlusTree.java:3987) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:317) > at > org.apache.ignite.interna
[jira] [Updated] (IGNITE-15962) CacheEntryRemovedListener makes the PUT operation to create ENTRY CREATED events
[ https://issues.apache.org/jira/browse/IGNITE-15962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15962: -- Remaining Estimate: 72h Original Estimate: 72h > CacheEntryRemovedListener makes the PUT operation to create ENTRY CREATED > events > > > Key: IGNITE-15962 > URL: https://issues.apache.org/jira/browse/IGNITE-15962 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: CacheContinuousQueryEntry-HeapDump.png, > CacheContinuousQueryEntry-StackTrace.png, EventListenerHeapUsage.java, > IgniteServer.java, build.gradle > > Original Estimate: 72h > Remaining Estimate: 72h > > {{CacheEntryRemovedListener}} configured with > {{MutableCacheEntryListenerConfiguration.isOldValueRequired = false}} makes > the {{Cache#put}} operation to create {{CREATED}} events on the primary > server for a short period of time. This is inefficient when working with > large objects: our application running with limited heap memory fails with > {{java.lang.OutOfMemoryError: Java heap space}} after executing a {{PUT}} > operation since the entry referencing the object is "stuck" on the heap. > h2. Reproducer > h3. Steps > We need two Ignite server nodes to demonstrate the problem. A cache is > deployed on both the nodes and both the nodes listen to the {{ENTRY REMOVED}} > events. Both the nodes disable fetching the {{CacheEntryEvent#getOldValue}} > by setting {{MutableCacheEntryListenerConfiguration.isOldValueRequired = > false}} > # Build and run the attached [^IgniteServer.java]. This server is configured > like described above and runs until manually terminated. > # Build the run the attached test [^EventListenerHeapUsage.java] . This test: > ## Starts another server configured like described above > ## Puts a large object taking 35% of the free memory. This is done in an > isolated scope so the memory after the operation is garbage collected, > ## Attempts to allocate 70% of the heap. > h3. Expected > The last "allocate 70% of the heap" operation should succeed. > h3. Actual > {noformat} > Caused by: java.lang.OutOfMemoryError: Java heap space > at > apache.ignite.issues.EventListenerHeapUsage.entryRemovedEventListenerFetchesValueOnHeap(EventListenerHeapUsage.java:33) > {noformat} > h2 Analysis > For some reason configuring the event listener makes the {{IgniteCache#put}} > operation to create and attach an instance of {{CacheContinuousQueryEntry}} > with {{evtType=CREATED}} and {{newObject}} referencing the entry. > The attached heap dump !CacheContinuousQueryEntry-HeapDump.png! collected on > the {{OutOfMemoryError}} demonstrates the large object "stuck" on the heap. > The attached {{CacheContinuousQueryEntry}} initialization > !CacheContinuousQueryEntry-StackTrace.png! proves the {{ENTRY CREATED}} event > is being created even though we did not configure it. > It seems wrong that Ignite needs to attached the {{ENTRY CREATED}} event to > the "update" message. Please enhance Ignite to not be doing that or explain > why Ignite must do that. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16137) ContainsKey operation fetches entry value into heap
[ https://issues.apache.org/jira/browse/IGNITE-16137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16137: -- Remaining Estimate: 168h Original Estimate: 168h > ContainsKey operation fetches entry value into heap > --- > > Key: IGNITE-16137 > URL: https://issues.apache.org/jira/browse/IGNITE-16137 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > [See similar problem for the remove() > operation|https://issues.apache.org/jira/browse/IGNITE-15959] > {{IgniteCache#containsKey(key)}} operation fetches full entry into heap > memory. This is inefficient when working with large objects: our application > running with limited heap memory fails with {{java.lang.OutOfMemoryError: > Java heap space}} when trying to check if a key exists. > It seems wrong that Ignite needs to fetch the full entry on heap to check if > the key exists. Please enhance Ignite to not be doing that or explain why > Ignite must do that. > h2. Reproducer > h3. Steps > Create a Gradle project with the below class and run it as > {{./gradlew test --tests apache.ignite.issues.ContainsOperationHeapUsage}} > {{build.gradle}} > {code:groovy} > test { > minHeapSize = "512m" > maxHeapSize = "512m" > } > {code} > {{ContainsOperationHeapUsage.java}} > {code:java} > public class ContainsOperationHeapUsage { > /** Run with -Xmx512m -Xms512m */ > @Test > public void containsOperationFetchesValueOnHeap() { > var igniteCfg = new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder(new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))) > ) > .setCacheConfiguration(new CacheConfiguration<>("blobs")); > try (var ignite = Ignition.start(igniteCfg)) { > Cache cache = ignite.cache("blobs"); > // Put a BLOB having size of 35% of free memory to the cache > Runtime.getRuntime().gc(); > var freeMemory = Runtime.getRuntime().freeMemory(); > var blobSize = (int)(freeMemory * 0.35); > putBlob(cache, blobSize); > // Use 70% of the free heap > Runtime.getRuntime().gc(); > var unused = new byte[2 * blobSize]; > // Check if the blob exists in the cache. > // This throws "OutOfMemoryError: Java heap space" since Ignite > retrieves full entry to the heap. > // Why does Ignite retrieve entry value to check if the key > exists? > cache.containsKey(1); > } > } > private static void putBlob(Cache cache, int blobSize) { > var blob = new byte[blobSize]; > cache.put(1, blob); > } > } > {code} > h3. Expected > The test passes > h3. Actual > The {{cache.containsKey}} operatoin fails with: > {noformat} > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1385) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:680) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:500) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:411) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:316) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:129) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:422) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetOne.found(BPlusTree.java:3156) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$
[jira] [Updated] (IGNITE-16272) IgniteOutOfMemoryException is not a critical failure
[ https://issues.apache.org/jira/browse/IGNITE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16272: -- Remaining Estimate: 168h Original Estimate: 168h > IgniteOutOfMemoryException is not a critical failure > > > Key: IGNITE-16272 > URL: https://issues.apache.org/jira/browse/IGNITE-16272 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > h3. The Problem > Ignite raises the > [IgniteOutOfMemoryException|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/internal/mem/IgniteOutOfMemoryException.java] > if a data region size is exceeded when trying to add more data to a cache. > Ignite considers the {{IgniteOutOfMemoryException}} as a critical failure. > This causes shutting down the Ignite server with the default failure handler. > However, reaching the data region capacity does not seem to be such a > critical problem requiring the server shutdown or restart. For example, in > our application we just want to report this problem back to the users and > notify the DevOps without applying the critical failure handler. To achieve > that, we had to define a custom {{FailureHandler}} that detects and ignores > the {{IgniteOutOfMemoryException}} and all the exception caused by the > {{{}IgniteOutOfMemoryException{}}}, allowing the final exception to reach the > application. This solution is clumsy and unreliable since it uses the > internal {{IgniteOutOfMemoryException}} definition and relies on a complex > secondary exception structure trying to find the > {{IgniteOutOfMemoryException}} among the suppressed exception and causes. > Ignite out-of-the-box failure handlers have the {{ignoredFailure}} property > that allows filtering out some kinds of failures. However, the > {{IgniteOutOfMemoryException}} is not among the > [FailureType|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/failure/FailureType.java] > that can be ignored. > h3. The Proposal > # Does anyone really want to treat the "data region capacity exceeded" > problem as a critical failure and stop or restart the server? > ** Consider never treating this condition as a critical failure. This change > is not backward compatible. > ** Or add another item to the {{FailureType}} enumeration to optionally > allow the users not to have that treated as a critical failure. This is > backward-compatible. > # Make the {{IgniteOutOfMemoryException}} a public API (now it is in the > {{internal}} package) > # Consider renaming {{IgniteOutOfMemoryException}} (for example, to > something like {{{}NotEnoughStorageException{}}}) since the current name is > similar to a really critical and usually unrecoverable Java's > {{OutOfMemoryError}} although the {{IgniteOutOfMemoryException}} is not that > critical. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16135) Expiry policy is not applied to Ignite cache after serialization
[ https://issues.apache.org/jira/browse/IGNITE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16135: -- Remaining Estimate: 168h Original Estimate: 168h > Expiry policy is not applied to Ignite cache after serialization > > > Key: IGNITE-16135 > URL: https://issues.apache.org/jira/browse/IGNITE-16135 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.11 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Expiry Policy specified in Ignite cache operation context is not applied to > the cache after the cache serialization/deserialization. > h3. Steps to Reproduce > # Run an Ignite server with a cache without expiry policy > # A client node joins the server, gets the cache proxy and applies expiry > policy to the cache proxy > # The client executes a Compute task taking the cache proxy as a parameter > on the server. > # The Compute task puts an entry to the cache using the cache proxy received > from the client. > # The client sleeps for a time period longer than the TTL specified for the > expiry policy > # The client checks if the entry that the Compute task added still exists > h3. Expected > The entry does not exist since it must expire > h3. Actual > The entry still exists in the cache. > h3. Reproducer > {code:java} > public class IgniteCacheSerializationTest { > /** Expiry Policy specified in Ignite cache operation context is enabled > after the cache serialization. */ > @Test > public void igniteCacheWithExpiryPolicySerialization() throws > InterruptedException { > var cacheName = "cache1"; > var ttl = 1000; > Supplier igniteCfgFactory = () -> > new IgniteConfiguration() > .setDiscoverySpi( > new TcpDiscoverySpi() > .setIpFinder( > new > TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500")) > ) > ) > .setCacheConfiguration(new CacheConfiguration<>(cacheName)); > try (var ignored = > Ignition.start(igniteCfgFactory.get().setIgniteInstanceName("server"))) { > try (var ignite = Ignition.start( > > igniteCfgFactory.get().setIgniteInstanceName("test").setClientMode(true)) > ) { > // Apply a 1-second expiry policy to the cache > var expiryPolicy = TouchedExpiryPolicy.factoryOf(new > Duration(TimeUnit.MILLISECONDS, ttl)).create(); > var cache = ignite. String>cache(cacheName).withExpiryPolicy(expiryPolicy); > // Run a Compute Task that serializes the cache with the > 1-second expiry policy applied. > // The task adds a key 1 to the cache > ignite.compute().affinityRun(cacheName, 1, new Put1(cache)); > // Sleep for a time twice longer than the TTL > Thread.sleep(ttl * 2); > // The key 1 must expiry by this time > assertFalse(cache.containsKey(1), "The entry still exists in > the cache"); > } > } > } > private static final class Put1 implements IgniteRunnable { > private final IgniteCache cache; > public Put1(IgniteCache cache) { > this.cache = cache; > } > @Override public void run() { > cache.put(1, "ignored"); > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Remaining Estimate: 168h Original Estimate: 168h > SSLException: closing inbound before receiving peer's close_notify > -- > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. Analysis > The problem _probably_ happens due to [this > code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] > calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} > alert, which TLS 1.2 is expecting (see [RFC > 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]). I guess the > right approach to close an SSL socket is just calling {{Socke#close}}, which > should properly wait/send a {{close_notify}} > Some references to consider: > [JDK-8215102 Closing connection to Mysql database results in > exception|https://bugs.openjdk.java.net/browse/JDK-8215102] > [Fix for #93590 - ignore javax.net.ssl.SSLException: closing inbound before > receiving peer's close_notify on > java11+|https://github.com/mysql/mysql-connector-j/pull/32] > [JDK-8251553 Socket closure issues in migrating from JDK 8 to JDK > 11|https://bugs.openjdk.java.net/browse/JDK-8251553] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16431) Entry expiration requires twice the entry size of heap
[ https://issues.apache.org/jira/browse/IGNITE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16431: -- Remaining Estimate: 168h Original Estimate: 168h > Entry expiration requires twice the entry size of heap > -- > > Key: IGNITE-16431 > URL: https://issues.apache.org/jira/browse/IGNITE-16431 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: 500MB-put-expiry-master.png > > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite takes twice the entry size off the heap to expire the entry when > {{{}eagerTtl=true{}}}. See the attached heap memory usage diagram of putting > and then expiring a 500MB entry in Ignite. > This makes Ignite inefficient with handling large objects causing > {{OutOfMemory}} errors. > Do we really need loading entry's value on heap at all to expiry the entry? > Please enhance Ignite cache entry expiration not to load the entry's value on > heap even once or explain why it is not possible. > !500MB-put-expiry-master.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16448) Optimize unacknowledged messages queue for FULL_SYNC caches
[ https://issues.apache.org/jira/browse/IGNITE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16448: -- Remaining Estimate: 168h Original Estimate: 168h > Optimize unacknowledged messages queue for FULL_SYNC caches > --- > > Key: IGNITE-16448 > URL: https://issues.apache.org/jira/browse/IGNITE-16448 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Attachments: apache-ignite-msgReqs-heap-usage.png > > Original Estimate: 168h > Remaining Estimate: 168h > > Apache Ignite queues all unacknowledged messages in the > {{GridNioRecoveryDescriptor#msgReqs}} queue. Size of this queue is > effectively controlled by the {{TcpCommunicationSpi.ackSendThreshold}} > property, which is 32 by default. > Queuing PUTs of large entries results in significant heap memory usage. > Please check if we really need to queue PUTs into FULL_SYNC caches. It seems > to me we should not since a failure to write to a backup would be sent back > to the client right away without any retries. If that is the case then please > optimize the heap usage by not queueing FULL_SYNC cache update messages. > See the heap usage example attached: !apache-ignite-msgReqs-heap-usage.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16715) Ignite should always use IgniteLogger
[ https://issues.apache.org/jira/browse/IGNITE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16715: -- Remaining Estimate: 168h Original Estimate: 168h > Ignite should always use IgniteLogger > - > > Key: IGNITE-16715 > URL: https://issues.apache.org/jira/browse/IGNITE-16715 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite sometimes logs directly to stdout (e.g. some startup output). This > makes it hard to setup custom logging. > Please make Ignite always use {{IgniteLogger}} or explain why it cannot be > done. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16889) Error-prone X#searchForCause() implementation
[ https://issues.apache.org/jira/browse/IGNITE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16889: -- Remaining Estimate: 24h Original Estimate: 24h > Error-prone X#searchForCause() implementation > - > > Key: IGNITE-16889 > URL: https://issues.apache.org/jira/browse/IGNITE-16889 > Project: Ignite > Issue Type: Bug > Components: general >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 24h > Remaining Estimate: 24h > > The {{X#seachForCause}} utility method fails with a {{NullPointerException}} > if the analyzed {{Throwable}} has no {{suppressed}} exception due to the > [missing null > check|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/typedef/X.java#L570]. > I never encountered any issues with that with real usage of Ignite. However, > writing a unit test that stabs the {{IgniteCache}} methods to throw an > exception without initialized {{suppressed}} exceptions results in the > stabbed method to return {{null}} instead of throwing an exception. The > workaround is to properly initialize the {{suppressed}} list in the mocked > exception. > I suggest to make the generic {{X#seachForCause}} utility method handle the > case when the {{suppressed}} list is not initialized. That would simplify the > unit tests code. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16716) Ignite JUL Logger should not change the configuration
[ https://issues.apache.org/jira/browse/IGNITE-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16716: -- Remaining Estimate: 168h Original Estimate: 168h > Ignite JUL Logger should not change the configuration > - > > Key: IGNITE-16716 > URL: https://issues.apache.org/jira/browse/IGNITE-16716 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > Ignite's {{JavaLogger}} forcefully re-configures logging by adding/removing > appenders and changing severity based on some system properties and > environment variables. > This makes it hard or even impossible to consistently switch between > different loggers keeping the same level of output. > Please enhance the {{JavaLogger}} to no mess with the configuration or > explain why that cannot be done -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (IGNITE-16922) Getting an entry with expiry policy causes IgniteOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16922: -- Remaining Estimate: 168h Original Estimate: 168h > Getting an entry with expiry policy causes IgniteOutOfMemoryException > - > > Key: IGNITE-16922 > URL: https://issues.apache.org/jira/browse/IGNITE-16922 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.13 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > Original Estimate: 168h > Remaining Estimate: 168h > > {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if > {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the > {{key}} and Ignite has not enough storage for another entry of the same or > bigger size. > This happens because: > # Ignite needs to update TTL > # TTL is part of the entry and Ignite overwrites full entry to update the TTL > # The problem is Ignite runs common code that checks if Ignite has enough > storage to write the entry with updated TTL back. The check fails causing the > {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}. > # This behavior is very confusing for Ignite users: why would a "read" > operation throw Ignite OOM? > Can we update the TTL atomically and skip the storage size check? > Please enhance Ignite not to throw Ignite OOM on {{get}}. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IGNITE-16922) Getting an entry with expiry policy causes IgniteOutOfMemoryException
Alexey Kukushkin created IGNITE-16922: - Summary: Getting an entry with expiry policy causes IgniteOutOfMemoryException Key: IGNITE-16922 URL: https://issues.apache.org/jira/browse/IGNITE-16922 Project: Ignite Issue Type: Bug Affects Versions: 2.13 Reporter: Alexey Kukushkin {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the {{key}} and Ignite has not enough storage for another entry of the same or bigger size. This happens because: # Ignite needs to update TTL # TTL is part of the entry and Ignite overwrites full entry to update the TTL # The problem is Ignite runs common code that checks if Ignite has enough storage to write the entry with updated TTL back. The check fails causing the {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}. # This behavior is very confusing for Ignite users: why would a "read" operation throw Ignite OOM? Can we update the TTL atomically and skip the storage size check? Please enhance Ignite not to throw Ignite OOM on {{get}}. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IGNITE-16889) Error-prone X#searchForCause() implementation
Alexey Kukushkin created IGNITE-16889: - Summary: Error-prone X#searchForCause() implementation Key: IGNITE-16889 URL: https://issues.apache.org/jira/browse/IGNITE-16889 Project: Ignite Issue Type: Bug Components: general Affects Versions: 2.12 Reporter: Alexey Kukushkin The {{X#seachForCause}} utility method fails with a {{NullPointerException}} if the analyzed {{Throwable}} has no {{suppressed}} exception due to the [missing null check|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/typedef/X.java#L570]. I never encountered any issues with that with real usage of Ignite. However, writing a unit test that stabs the {{IgniteCache}} methods to throw an exception without initialized {{suppressed}} exceptions results in the stabbed method to return {{null}} instead of throwing an exception. The workaround is to properly initialize the {{suppressed}} list in the mocked exception. I suggest to make the generic {{X#seachForCause}} utility method handle the case when the {{suppressed}} list is not initialized. That would simplify the unit tests code. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IGNITE-16716) Ignite JUL Logger should not change the configuration
Alexey Kukushkin created IGNITE-16716: - Summary: Ignite JUL Logger should not change the configuration Key: IGNITE-16716 URL: https://issues.apache.org/jira/browse/IGNITE-16716 Project: Ignite Issue Type: Improvement Reporter: Alexey Kukushkin Ignite's {{JavaLogger}} forcefully re-configures logging by adding/removing appenders and changing severity based on some system properties and environment variables. This makes it hard or even impossible to consistently switch between different loggers keeping the same level of output. Please enhance the {{JavaLogger}} to no mess with the configuration or explain why that cannot be done -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IGNITE-16715) Ignite should always use IgniteLogger
Alexey Kukushkin created IGNITE-16715: - Summary: Ignite should always use IgniteLogger Key: IGNITE-16715 URL: https://issues.apache.org/jira/browse/IGNITE-16715 Project: Ignite Issue Type: Improvement Reporter: Alexey Kukushkin Ignite sometimes logs directly to stdout (e.g. some startup output). This makes it hard to setup custom logging. Please make Ignite always use {{IgniteLogger}} or explain why it cannot be done. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Description: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting (see [RFC 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]). I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} Some references to consider: [JDK-8215102 Closing connection to Mysql database results in exception|https://bugs.openjdk.java.net/browse/JDK-8215102] [Fix for #93590 - ignore javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify on java11+|https://github.com/mysql/mysql-connector-j/pull/32] [JDK-8251553 Socket closure issues in migrating from JDK 8 to JDK 11|https://bugs.openjdk.java.net/browse/JDK-8251553] was: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting (see [RFC 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]). I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} > SSLException: closing inbound before receiving peer's close_notify > -- > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.igni
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Summary: SSLException: closing inbound before receiving peer's close_notify (was: SSLException: closing inbound before receiving peer's close_notify (TLS 1.2)) > SSLException: closing inbound before receiving peer's close_notify > -- > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. h3. Analysis > The problem _probably_ happens due to [this > code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] > calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} > alert, which TLS 1.2 is expecting (see [RFC > 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]). I guess the > right approach to close an SSL socket is just calling {{Socke#close}}, which > should properly wait/send a {{close_notify}} -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify (TLS 1.2)
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Description: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting (see [RFC 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]). I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} was: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting (see [RFC 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]. I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} > SSLException: closing inbound before receiving peer's close_notify (TLS 1.2) > > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. h3. Analysis > The problem
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify (TLS 1.2)
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Description: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting (see [RFC 8446|https://datatracker.ietf.org/doc/html/rfc8446#section-6]. I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} was: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting. I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} > SSLException: closing inbound before receiving peer's close_notify (TLS 1.2) > > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. h3. Analysis > The problem _probably_ happens due to [this > code|https://github.com/apache/ignite/
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify (TLS 1.2)
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Description: Ignite nodes output the warning below on startup when TLS protocol v1.2 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.2 is expecting. I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} was: Ignite nodes output the warning below on startup when TLS protocol v1.3 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.3 is expecting. I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} > SSLException: closing inbound before receiving peer's close_notify (TLS 1.2) > > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > Ignite nodes output the warning below on startup when TLS protocol v1.2 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. h3. Analysis > The problem _probably_ happens due to [this > code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/Ig
[jira] [Updated] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify (TLS 1.2)
[ https://issues.apache.org/jira/browse/IGNITE-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16496: -- Summary: SSLException: closing inbound before receiving peer's close_notify (TLS 1.2) (was: SSLException: closing inbound before receiving peer's close_notify (TLS 1.3)) > SSLException: closing inbound before receiving peer's close_notify (TLS 1.2) > > > Key: IGNITE-16496 > URL: https://issues.apache.org/jira/browse/IGNITE-16496 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.12 >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > Ignite nodes output the warning below on startup when TLS protocol v1.3 is > used: > {noformat} > 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] > o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing > inbound before receiving peer's close_notify > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) > ~[na:na] >at > java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) > ~[na:na] >at > org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at > org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) > ~[ignite-core-2.12.0.jar!/:2.12.0] >at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} > To reproduce the problem just start two server nodes with TLS v1.3 enabled > and the warnings will be printed in the log before the cluster is formed. > h3. h3. Analysis > The problem _probably_ happens due to [this > code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] > calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} > alert, which TLS 1.3 is expecting. I guess the right approach to close an SSL > socket is just calling {{Socke#close}}, which should properly wait/send a > {{close_notify}} -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IGNITE-16496) SSLException: closing inbound before receiving peer's close_notify (TLS 1.3)
Alexey Kukushkin created IGNITE-16496: - Summary: SSLException: closing inbound before receiving peer's close_notify (TLS 1.3) Key: IGNITE-16496 URL: https://issues.apache.org/jira/browse/IGNITE-16496 Project: Ignite Issue Type: Bug Affects Versions: 2.12 Reporter: Alexey Kukushkin Ignite nodes output the warning below on startup when TLS protocol v1.3 is used: {noformat} 2022-02-08 11:53:05.705 WARN 19384 --- [1:62095]-#4-#51] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:745) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:724) ~[na:na] at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4249) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7370) ~[ignite-core-2.12.0.jar!/:2.12.0] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) ~[ignite-core-2.12.0.jar!/:2.12.0] {noformat} To reproduce the problem just start two server nodes with TLS v1.3 enabled and the warnings will be printed in the log before the cluster is formed. h3. h3. Analysis The problem _probably_ happens due to [this code|https://github.com/apache/ignite/blob/2.12.0/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L4426] calling {{Socket#shutdownInput()}} before receiving SSL {{close_notify}} alert, which TLS 1.3 is expecting. I guess the right approach to close an SSL socket is just calling {{Socke#close}}, which should properly wait/send a {{close_notify}} -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IGNITE-16448) Optimize unacknowledged messages queue for FULL_SYNC caches
Alexey Kukushkin created IGNITE-16448: - Summary: Optimize unacknowledged messages queue for FULL_SYNC caches Key: IGNITE-16448 URL: https://issues.apache.org/jira/browse/IGNITE-16448 Project: Ignite Issue Type: Improvement Affects Versions: 2.12 Reporter: Alexey Kukushkin Attachments: apache-ignite-msgReqs-heap-usage.png Apache Ignite queues all unacknowledged messages in the {{GridNioRecoveryDescriptor#msgReqs}} queue. Size of this queue is effectively controlled by the {{TcpCommunicationSpi.ackSendThreshold}} property, which is 32 by default. Queuing PUTs of large entries results in significant heap memory usage. Please check if we really need to queue PUTs into FULL_SYNC caches. It seems to me we should not since a failure to write to a backup would be sent back to the client right away without any retries. If that is the case then please optimize the heap usage by not queueing FULL_SYNC cache update messages. See the heap usage example attached: !apache-ignite-msgReqs-heap-usage.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IGNITE-16431) Entry expiration requires twice the entry size of heap
Alexey Kukushkin created IGNITE-16431: - Summary: Entry expiration requires twice the entry size of heap Key: IGNITE-16431 URL: https://issues.apache.org/jira/browse/IGNITE-16431 Project: Ignite Issue Type: Improvement Affects Versions: 2.12 Reporter: Alexey Kukushkin Attachments: 500MB-put-expiry-master.png Ignite takes twice the entry size off the heap to expire the entry when {{{}eagerTtl=true{}}}. See the attached heap memory usage diagram of putting and then expiring a 500MB entry in Ignite. This makes Ignite inefficient with handling large objects causing {{OutOfMemory}} errors. Do we really need loading entry's value on heap at all to expiry the entry? Please enhance Ignite cache entry expiration not to load the entry's value on heap even once or explain why it is not possible. !500MB-put-expiry-master.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (IGNITE-16272) IgniteOutOfMemoryException is not a critical failure
[ https://issues.apache.org/jira/browse/IGNITE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16272: -- Labels: cggg (was: c# cggg) > IgniteOutOfMemoryException is not a critical failure > > > Key: IGNITE-16272 > URL: https://issues.apache.org/jira/browse/IGNITE-16272 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: cggg > > h3. The Problem > Ignite raises the > [IgniteOutOfMemoryException|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/internal/mem/IgniteOutOfMemoryException.java] > if a data region size is exceeded when trying to add more data to a cache. > Ignite considers the {{IgniteOutOfMemoryException}} as a critical failure. > This causes shutting down the Ignite server with the default failure handler. > However, reaching the data region capacity does not seem to be such a > critical problem requiring the server shutdown or restart. For example, in > our application we just want to report this problem back to the users and > notify the DevOps without applying the critical failure handler. To achieve > that, we had to define a custom {{FailureHandler}} that detects and ignores > the {{IgniteOutOfMemoryException}} and all the exception caused by the > {{{}IgniteOutOfMemoryException{}}}, allowing the final exception to reach the > application. This solution is clumsy and unreliable since it uses the > internal {{IgniteOutOfMemoryException}} definition and relies on a complex > secondary exception structure trying to find the > {{IgniteOutOfMemoryException}} among the suppressed exception and causes. > Ignite out-of-the-box failure handlers have the {{ignoredFailure}} property > that allows filtering out some kinds of failures. However, the > {{IgniteOutOfMemoryException}} is not among the > [FailureType|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/failure/FailureType.java] > that can be ignored. > h3. The Proposal > # Does anyone really want to treat the "data region capacity exceeded" > problem as a critical failure and stop or restart the server? > ** Consider never treating this condition as a critical failure. This change > is not backward compatible. > ** Or add another item to the {{FailureType}} enumeration to optionally > allow the users not to have that treated as a critical failure. This is > backward-compatible. > # Make the {{IgniteOutOfMemoryException}} a public API (now it is in the > {{internal}} package) > # Consider renaming {{IgniteOutOfMemoryException}} (for example, to > something like {{{}NotEnoughStorageException{}}}) since the current name is > similar to a really critical and usually unrecoverable Java's > {{OutOfMemoryError}} although the {{IgniteOutOfMemoryException}} is not that > critical. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (IGNITE-16272) IgniteOutOfMemoryException is not a critical failure
[ https://issues.apache.org/jira/browse/IGNITE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-16272: -- Labels: c# cggg (was: cggg) > IgniteOutOfMemoryException is not a critical failure > > > Key: IGNITE-16272 > URL: https://issues.apache.org/jira/browse/IGNITE-16272 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Kukushkin >Priority: Major > Labels: c#, cggg > > h3. The Problem > Ignite raises the > [IgniteOutOfMemoryException|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/internal/mem/IgniteOutOfMemoryException.java] > if a data region size is exceeded when trying to add more data to a cache. > Ignite considers the {{IgniteOutOfMemoryException}} as a critical failure. > This causes shutting down the Ignite server with the default failure handler. > However, reaching the data region capacity does not seem to be such a > critical problem requiring the server shutdown or restart. For example, in > our application we just want to report this problem back to the users and > notify the DevOps without applying the critical failure handler. To achieve > that, we had to define a custom {{FailureHandler}} that detects and ignores > the {{IgniteOutOfMemoryException}} and all the exception caused by the > {{{}IgniteOutOfMemoryException{}}}, allowing the final exception to reach the > application. This solution is clumsy and unreliable since it uses the > internal {{IgniteOutOfMemoryException}} definition and relies on a complex > secondary exception structure trying to find the > {{IgniteOutOfMemoryException}} among the suppressed exception and causes. > Ignite out-of-the-box failure handlers have the {{ignoredFailure}} property > that allows filtering out some kinds of failures. However, the > {{IgniteOutOfMemoryException}} is not among the > [FailureType|https://github.com/apache/ignite/blob/2.11.1/modules/core/src/main/java/org/apache/ignite/failure/FailureType.java] > that can be ignored. > h3. The Proposal > # Does anyone really want to treat the "data region capacity exceeded" > problem as a critical failure and stop or restart the server? > ** Consider never treating this condition as a critical failure. This change > is not backward compatible. > ** Or add another item to the {{FailureType}} enumeration to optionally > allow the users not to have that treated as a critical failure. This is > backward-compatible. > # Make the {{IgniteOutOfMemoryException}} a public API (now it is in the > {{internal}} package) > # Consider renaming {{IgniteOutOfMemoryException}} (for example, to > something like {{{}NotEnoughStorageException{}}}) since the current name is > similar to a really critical and usually unrecoverable Java's > {{OutOfMemoryError}} although the {{IgniteOutOfMemoryException}} is not that > critical. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (IGNITE-15241) Ignite H2 Security Vulnerabilities
[ https://issues.apache.org/jira/browse/IGNITE-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kukushkin updated IGNITE-15241: -- Labels: cggg (was: ) > Ignite H2 Security Vulnerabilities > -- > > Key: IGNITE-15241 > URL: https://issues.apache.org/jira/browse/IGNITE-15241 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.10 >Reporter: Alexey Kukushkin >Assignee: Alexey Kukushkin >Priority: Major > Labels: cggg > > Upgrade H2 dependency of the ignite-indexing module to the latest version > 1.4.200. > Apache Ignite SQL (module {{ignite-indexing}}) depends on H2 database version > 1.4.197, which has these two [security > vulnerabilities|https://www.cvedetails.com/vulnerability-list/vendor_id-17893/product_id-45580/year-2018/H2database-H2.html] > [CVE-2018-14335|https://www.cvedetails.com/cve/CVE-2018-14335/] is regarded > as a critical vulnerability by our analyzer (Black Duck SCA) and makes it > impossible to use Ignite SQL due to security policies. We realize this > vulnerability is probably not even applicable to the H2 in Ignite since there > is no H2 database or H2 backups in Ignite. Still the security policies are > very formal and do not allow that anyway. > We believe there are lots of other enterprises having the same issue. For > example, there is another issue IGNITE-14381 referencing the same problem. > The latest H2 1.4.200 has no vulnerabilities. -- This message was sent by Atlassian Jira (v8.20.1#820001)