[jira] [Commented] (IGNITE-18826) Java thin client hungs on close
[ https://issues.apache.org/jira/browse/IGNITE-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690633#comment-17690633 ] Ignite TC Bot commented on IGNITE-18826: {panel:title=Branch: [pull/10549/head] Base: [master] : No blockers found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel} {panel:title=Branch: [pull/10549/head] Base: [master] : New Tests (2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1} {color:#8b}Thin Client: Java{color} [[tests 2|https://ci2.ignite.apache.org/viewLog.html?buildId=7054497]] * {color:#013220}ClientTestSuite: ThinClientPartitionAwarenessUnstableTopologyTest.testSessionCloseBeforeHandshake - PASSED{color} * {color:#013220}ClientTestSuite: ThinClientPartitionAwarenessUnstableTopologyTest.testCreateSessionAfterClose - PASSED{color} {panel} [TeamCity *-- Run :: All* Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7054511buildTypeId=IgniteTests24Java8_RunAll] > Java thin client hungs on close > --- > > Key: IGNITE-18826 > URL: https://issues.apache.org/jira/browse/IGNITE-18826 > Project: Ignite > Issue Type: Bug >Reporter: Nikita Amelchev >Assignee: Nikita Amelchev >Priority: Major > Labels: ise > Fix For: 2.15 > > Time Spent: 10m > Remaining Estimate: 0h > > There are several places where thin client threads can hang: > GridNioClientConnectionMultiplexer: SSL enabled + PA enabled > {noformat} > // Can return finished future with error -> sslHandshakeFut will not complete. > GridNioFuture sesFut = srv.createSession(ch, meta, false, > null); > if (sslHandshakeFut != null) > sslHandshakeFut.get(); > {noformat} > TcpClientChannel: PA enabled > {noformat} > ClientRequestFuture fut = new ClientRequestFuture(requestId, > ClientOperation.HANDSHAKE); > // TcpClientChannel may be closed and fut will not complete. > assert !closed.get(); > pendingReqs.put(requestId, fut); > ... > ByteBuffer buf = timeout > 0 ? fut.get(timeout) : fut.get(); > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18640) Implement placement driver best-effort single actor selector and fail-over
[ https://issues.apache.org/jira/browse/IGNITE-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-18640: -- Reviewer: Vladislav Pyatkov > Implement placement driver best-effort single actor selector and fail-over > -- > > Key: IGNITE-18640 > URL: https://issues.apache.org/jira/browse/IGNITE-18640 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Assignee: Denis Chudov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > h3. Motivation > As a prerequisite, it's worth to mention that placement drive itself should > be reliable and have corresponding fail-over logic, meaning that placement > driver service should be distributed in a way that if one of its nodes fails > another one picks up the flag. On the other hand, despite the fact, that it's > valid to have more than one PD active actors (the one that will check > topology, send leaseGrant msg, etc) it's better to have one only in order to > reduce the amount of unnecessary calculations, messaging duplication and so > on. So, to sum up: > * PD may work on top of meta storage, using it as a consensus provider. > * There may be more than one active PD actors, that try to evaluate primary > replica along with corresponding lease, send leaseGrant msg, etc, meaning > that actions should be idempotent or that we should have an ability to skip > stale/concurrent triggers. > * It worth to have at least best-effort single actor selection logic. > h3. Definition of Done > * Almost always (because of best-effort nature, it's not always) there's > only one PD active actor if there's a majority in ms group. > * If for some reason active actor fails, another one will picks up the flag > as fast as possible. > * It's still valid to have multiple active actors at the same time. If you > guys have any ideas of how to implement not more than one actor, please share > them. > h3. Implementation Notes > Assuming that we have a distributed onLeaderElected(Peer leader, long term) > callback we may implement following logic on PlacementDriverManager#start() > * register ms.onLeaderElected() > {code:java} > ms.onLeaderElected((leader, term) -> { > if (term > lastSeenTerm) { > if (leader.equlas(localNode)) { > // Become an active actor. > } else { > // Discard activeness. > } > } else { > // No-op, just a stale update. > } > });{code} > * refreshLeader and to exact the same logic as the one mentioned above in > order to become and active actor if there already was a leader during > listener registration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-15322) System tasks should run without any explicitly granted permissions
[ https://issues.apache.org/jira/browse/IGNITE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690583#comment-17690583 ] Mikhail Petrov commented on IGNITE-15322: - [~av] Thank you a lot for the review! > System tasks should run without any explicitly granted permissions > -- > > Key: IGNITE-15322 > URL: https://issues.apache.org/jira/browse/IGNITE-15322 > Project: Ignite > Issue Type: Bug > Components: compute, security >Reporter: Ilya Kazakov >Assignee: Mikhail Petrov >Priority: Minor > Labels: ise > Time Spent: 2h 20m > Remaining Estimate: 0h > > For example, this code needs TASK_EXECUTE permissions. > {code:java} > Affinity affinity = ignite.affinity("TEST"); > affinity.mapKeysToNodes(Arrays.asList(1L,100L, 1000L));{code} > This is unexpected behavior, because: > - the task started implicitly (under the hood), customer should not to know > about it. > - this is a system task (not defined by a customer), the tasks needs for a > normal grid workflow. > Also, I suppose there are any other implicitly tasks, which could lead to > unexpected behavior (need permissions). > Proposed way to solve this issue: > 1. Add mechanism to destinguish whether task class is SYSTEM (part of the > Ignite codebase) or USER. > Here we can reuse SecurityUtils#isSystemType mechanism that is used in Ignite > Sanbox implementation. > 2. Add mechanism to detect if task execution was initiated by the user > (PUBLIC CALL) or by the Ignite system itself (INTERNAL CALL). > It seems that the easiest way to achieve this is to completely separate the > public and private Compute APIs. > Task executioin requests received through Ignite Thin Clients are considered > PUBLIC CALLs. > The first two steps give us the ability to > A. safely skip authorization of SYSTEM tasks which are > called through INTERNAL API. > B. keep authorization of PUBLIC tasks intact > C. prevent users of calling SYSTEM tasks directly through PUBLIC API ( it > means that all user task execution requests received through REST or Thin > client protocols MUST be executed through PUBLIC API). > 3. Add the ability to explicitly specify for SYSTEM > task/callable/runnable/closure what permission should be checked before its > execution. > It can be solved by introducing optionsl interface that compute job can > implement. > {code:java} > /** */ > public interface SecurityAwareJob { > /** */ > public SecurityPermissionSet requiredPermissions(); > } > {code} > 4. SYSTEM tasks can splitted into two categories - > SYSTEM INTERNAL (tasks that are not available to the user) and SYSTEM PUBLIC > tasks (tasks that are part oof the ignite code but are available to the user > and can be executed through the PUBLIC API) > Example of SYSTEM public tasks - Visor tasks on which the user control script > is implemented > They are executed through Thin Client which is considered Public API. > Considering that SYSTEM PUBLIC tasks can potentially be executed by the user, > we must force the developer to explicitly specify permissions for tasks of > this type. It can be done by checking that SYSTEM tasks that are executed > through PUBLIC API impelements PublicAwareJob interface described above. > ||X||Public API||Private API|| > |PUBLIC task|auth by task name|restricted| > |SYSTEM INTERNAL task|restricted|auth skipped| > |SYSTEM PUBLIC task|auth by explicitly specified permissions|auth by > explicitly specified permissions| > By the way the SecurityAwareJob interface, implemented by all Visor jobs, > allows you to specify for each specific Visor job what permissions it > requires. Also it is proposed to guard all Visor Jobs with ADMIN_OPS > permission by default. > 5. Authorization of SYSTEM tasks cancellation must be skipped it they are > canceled by the same user who started them, oterwise dedicated permissions is > required (e.g. ADMIN_KILL). USER tasks cancellation is performed by their > names. > Possible troubles: > 1. Mentioned SecurityUtils#isSystemType works only for the ignite-core > module. If some tasks are defined inside other Ignite modules - they will not > be considered SYSTEM. Currently there are no such task. > 2. Currently all DotNet code is wrapped and executed via special SYSTEM > tasks. As a result all DotNet runnable/callable/tasks are authorized by the > name of the SYSTEM wrapper task. So if TASK_EXECUTE permissions for the > SYSTEM wrapper task is granted - user can execute whathever DotNet custom tas > he wants. After the propsed changes SYTEM task authorization will no more.be > performed. So DotNet custom code execution will become completeley > unauthorised. We should fix it in a separate ticket. -- This message was sent by Atlassian Jira
[jira] [Updated] (IGNITE-15322) System tasks should run without any explicitly granted permissions
[ https://issues.apache.org/jira/browse/IGNITE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Petrov updated IGNITE-15322: Description: For example, this code needs TASK_EXECUTE permissions. {code:java} Affinity affinity = ignite.affinity("TEST"); affinity.mapKeysToNodes(Arrays.asList(1L,100L, 1000L));{code} This is unexpected behavior, because: - the task started implicitly (under the hood), customer should not to know about it. - this is a system task (not defined by a customer), the tasks needs for a normal grid workflow. Also, I suppose there are any other implicitly tasks, which could lead to unexpected behavior (need permissions). Proposed way to solve this issue: 1. Add mechanism to destinguish whether task class is SYSTEM (part of the Ignite codebase) or USER. Here we can reuse SecurityUtils#isSystemType mechanism that is used in Ignite Sanbox implementation. 2. Add mechanism to detect if task execution was initiated by the user (PUBLIC CALL) or by the Ignite system itself (INTERNAL CALL). It seems that the easiest way to achieve this is to completely separate the public and private Compute APIs. Task executioin requests received through Ignite Thin Clients are considered PUBLIC CALLs. The first two steps give us the ability to A. safely skip authorization of SYSTEM tasks which are called through INTERNAL API. B. keep authorization of PUBLIC tasks intact C. prevent users of calling SYSTEM tasks directly through PUBLIC API ( it means that all user task execution requests received through REST or Thin client protocols MUST be executed through PUBLIC API). 3. Add the ability to explicitly specify for SYSTEM task/callable/runnable/closure what permission should be checked before its execution. It can be solved by introducing optionsl interface that compute job can implement. {code:java} /** */ public interface SecurityAwareJob { /** */ public SecurityPermissionSet requiredPermissions(); } {code} 4. SYSTEM tasks can splitted into two categories - SYSTEM INTERNAL (tasks that are not available to the user) and SYSTEM PUBLIC tasks (tasks that are part oof the ignite code but are available to the user and can be executed through the PUBLIC API) Example of SYSTEM public tasks - Visor tasks on which the user control script is implemented They are executed through Thin Client which is considered Public API. Considering that SYSTEM PUBLIC tasks can potentially be executed by the user, we must force the developer to explicitly specify permissions for tasks of this type. It can be done by checking that SYSTEM tasks that are executed through PUBLIC API impelements PublicAwareJob interface described above. ||X||Public API||Private API|| |PUBLIC task|auth by task name|restricted| |SYSTEM INTERNAL task|restricted|auth skipped| |SYSTEM PUBLIC task|auth by explicitly specified permissions|auth by explicitly specified permissions| By the way the SecurityAwareJob interface, implemented by all Visor jobs, allows you to specify for each specific Visor job what permissions it requires. Also it is proposed to guard all Visor Jobs with ADMIN_OPS permission by default. 5. Authorization of SYSTEM tasks cancellation must be skipped it they are canceled by the same user who started them, oterwise dedicated permissions is required (e.g. ADMIN_KILL). USER tasks cancellation is performed by their names. Possible troubles: 1. Mentioned SecurityUtils#isSystemType works only for the ignite-core module. If some tasks are defined inside other Ignite modules - they will not be considered SYSTEM. Currently there are no such task. 2. Currently all DotNet code is wrapped and executed via special SYSTEM tasks. As a result all DotNet runnable/callable/tasks are authorized by the name of the SYSTEM wrapper task. So if TASK_EXECUTE permissions for the SYSTEM wrapper task is granted - user can execute whathever DotNet custom tas he wants. After the propsed changes SYTEM task authorization will no more.be performed. So DotNet custom code execution will become completeley unauthorised. We should fix it in a separate ticket. was: For example, this code needs TASK_EXECUTE permissions. {code:java} Affinity affinity = ignite.affinity("TEST"); affinity.mapKeysToNodes(Arrays.asList(1L,100L, 1000L));{code} This is unexpected behavior, because: - the task started implicitly (under the hood), customer should not to know about it. - this is a system task (not defined by a customer), the tasks needs for a normal grid workflow. Also, I suppose there are any other implicitly tasks, which could lead to unexpected behavior (need permissions). Proposed way to solve this issue: 1. Add mechanism to destinguish whether task class is SYSTEM (part of the Ignite codebase) or USER. Here we can reuse SecurityUtils#isSystemType mechanism that is used in Ignite Sanbox implementation. 2. Add mechanism to
[jira] [Commented] (IGNITE-15322) System tasks should run without any explicitly granted permissions
[ https://issues.apache.org/jira/browse/IGNITE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690576#comment-17690576 ] Ignite TC Bot commented on IGNITE-15322: {panel:title=Branch: [pull/10226/head] Base: [master] : No blockers found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel} {panel:title=Branch: [pull/10226/head] Base: [master] : New Tests (10)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1} {color:#8b}Java Client{color} [[tests 6|https://ci2.ignite.apache.org/viewLog.html?buildId=7054334]] * {color:#013220}IgniteClientTestSuite: ComputeTaskPermissionsTest.testPublicAccessSystemTask - PASSED{color} * {color:#013220}IgniteClientTestSuite: ComputeTaskPermissionsTest.testSystemTaskCancel - PASSED{color} * {color:#013220}IgniteClientTestSuite: ComputeTaskPermissionsTest.testNode - PASSED{color} * {color:#013220}IgniteClientTestSuite: ComputeTaskPermissionsTest.testIgniteClient - PASSED{color} * {color:#013220}IgniteClientTestSuite: ComputeTaskPermissionsTest.testRestClient - PASSED{color} * {color:#013220}IgniteClientTestSuite: ComputeTaskPermissionsTest.testGridClient - PASSED{color} {color:#8b}Security{color} [[tests 4|https://ci2.ignite.apache.org/viewLog.html?buildId=7054370]] * {color:#013220}SecurityTestSuite: ServiceAuthorizationTest.testRemoteServiceInvoke[isClient=true] - PASSED{color} * {color:#013220}SecurityTestSuite: ServiceAuthorizationTest.testLocalServiceInvoke[isClient=true] - PASSED{color} * {color:#013220}SecurityTestSuite: ServiceAuthorizationTest.testRemoteServiceInvoke[isClient=false] - PASSED{color} * {color:#013220}SecurityTestSuite: ServiceAuthorizationTest.testLocalServiceInvoke[isClient=false] - PASSED{color} {panel} [TeamCity *-- Run :: All* Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7054394buildTypeId=IgniteTests24Java8_RunAll] > System tasks should run without any explicitly granted permissions > -- > > Key: IGNITE-15322 > URL: https://issues.apache.org/jira/browse/IGNITE-15322 > Project: Ignite > Issue Type: Bug > Components: compute, security >Reporter: Ilya Kazakov >Assignee: Mikhail Petrov >Priority: Minor > Labels: ise > Time Spent: 2h 10m > Remaining Estimate: 0h > > For example, this code needs TASK_EXECUTE permissions. > {code:java} > Affinity affinity = ignite.affinity("TEST"); > affinity.mapKeysToNodes(Arrays.asList(1L,100L, 1000L));{code} > This is unexpected behavior, because: > - the task started implicitly (under the hood), customer should not to know > about it. > - this is a system task (not defined by a customer), the tasks needs for a > normal grid workflow. > Also, I suppose there are any other implicitly tasks, which could lead to > unexpected behavior (need permissions). > Proposed way to solve this issue: > 1. Add mechanism to destinguish whether task class is SYSTEM (part of the > Ignite codebase) or USER. > Here we can reuse SecurityUtils#isSystemType mechanism that is used in Ignite > Sanbox implementation. > 2. Add mechanism to detect if task execution was initiated by the user > (PUBLIC CALL) or by the Ignite system itself (INTERNAL CALL). > It seems that the easiest way to achieve this is to completely separate the > public and private Compute APIs. > Task executioin requests received through Ignite Thin Clients are considered > PUBLIC CALLs. > The first two steps give us the ability to > A. safely skip authorization of SYSTEM tasks which are > called through INTERNAL API. > B. keep authorization of PUBLIC tasks intact > C. prevent users of calling SYSTEM tasks directly through PUBLIC API ( it > means that all user task execution requests received through REST or Thin > client protocols MUST be executed through PUBLIC API). > 3. Add the ability to explicitly specify for SYSTEM > task/callable/runnable/closure what permission should be checked before its > execution. > It can be solved by introducing optionsl interface that compute job can > implement. > {code:java} > /** */ > public interface SecurityAwareJob { > /** */ > public SecurityPermissionSet requiredPermissions(); > } > {code} > 4. SYSTEM tasks can splitted into two categories - > SYSTEM INTERNAL (tasks that are not available to the user) and SYSTEM PUBLIC > tasks (tasks that are part oof the ignite code but are available to the user > and can be executed through the PUBLIC API) > Examples of SYSTEM public tasks - > - Visor tasks on which the user control script is implemented > - JDBC tasks > All of them are executed through Thin Client which is considered Public API. > Considering that SYSTEM PUBLIC tasks can potentially be executed by the user, > we must force the developer to explicitly specify permissions for tasks
[jira] [Created] (IGNITE-18837) CacheStripedExecutor should handle error during execution of submitted task
Maksim Timonin created IGNITE-18837: --- Summary: CacheStripedExecutor should handle error during execution of submitted task Key: IGNITE-18837 URL: https://issues.apache.org/jira/browse/IGNITE-18837 Project: Ignite Issue Type: Sub-task Reporter: Maksim Timonin Assignee: Maksim Timonin Fix For: 2.15 Case: 4 nodes, 2 backups - some nodes don't have GridDhtLocalPartition instance. Currently it leads to NPE during finalizing counters, but IS swallows this error. This case doesn't corrupt data, but there are possible cases when it can happen. CacheStripedExecutor should handle such errors and mark operations as failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-12483) ReflectionFactory is essential thanks to PlatformDotNetSessionLockResult
[ https://issues.apache.org/jira/browse/IGNITE-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Plekhanov reassigned IGNITE-12483: -- Assignee: Aleksey Plekhanov > ReflectionFactory is essential thanks to PlatformDotNetSessionLockResult > > > Key: IGNITE-12483 > URL: https://issues.apache.org/jira/browse/IGNITE-12483 > Project: Ignite > Issue Type: Bug > Components: binary >Reporter: Ilya Kasnacheev >Assignee: Aleksey Plekhanov >Priority: Major > Labels: ise, usability > > We currently treat ReflectionFactory as a nice-to-have thing, so we silently > ignore failures of its reflection: > {code} > try { > Class refFactoryCls = > Class.forName("sun.reflect.ReflectionFactory"); > refFac = > refFactoryCls.getMethod("getReflectionFactory").invoke(null); > ctorFac = > refFac.getClass().getMethod("newConstructorForSerialization", Class.class, > Constructor.class); > } > catch (NoSuchMethodException | ClassNotFoundException | > IllegalAccessException | InvocationTargetException ignored) { > // No-op. > } > {code} > However, it is now essential thanks to the class > PlatformDotNetSessionLockResult, which is always registered during note > start-up and which does not have empty constructor. > So not having access to ReflectionFactory (JBoss will hide it, for example) > will lead to the following cryptic exception (courtesy stack overflow): > {code} > 2019-12-19 09:11:39,355 SEVERE [org.apache.ignite.internal.IgniteKernal] > (ServerService Thread Pool -- 81) Got exception while starting (will rollback > startup routine).: class org.apache.ignite.binary.BinaryObjectException: > Failed to find empty constructor for class: > org.apache.ignite.internal.processors.platform.websession.PlatformDotNetSessionLockResult > at > deployment.StreamsApp.ear//org.apache.ignite.internal.binary.BinaryClassDescriptor.constructor(BinaryClassDescriptor.java:981) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.binary.BinaryClassDescriptor.(BinaryClassDescriptor.java:267) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.binary.BinaryContext.registerPredefinedType(BinaryContext.java:1063) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.binary.BinaryContext.registerPredefinedType(BinaryContext.java:1048) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.binary.BinaryContext.(BinaryContext.java:350) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.start(CacheObjectBinaryProcessorImpl.java:208) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1700) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1013) > at > deployment.StreamsApp.ear//org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038) > at > org.jboss.as.ee@18.0.1.Final//org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88) > {code} > My suggestions are the following: > - Introduce a warning when ReflectionFactory not found instead of ignoring > exception. > - Add empty constructor to PlatformDotNetSessionLockResult and make sure no > other classes need reflection during start-up. > - (optionally) instead, introduce an error when ReflectionFactory not found. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18826) Java thin client hungs on close
[ https://issues.apache.org/jira/browse/IGNITE-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Amelchev updated IGNITE-18826: - Release Note: Java thin client: fixed the pending requests race on close > Java thin client hungs on close > --- > > Key: IGNITE-18826 > URL: https://issues.apache.org/jira/browse/IGNITE-18826 > Project: Ignite > Issue Type: Bug >Reporter: Nikita Amelchev >Assignee: Nikita Amelchev >Priority: Major > Labels: ise > Fix For: 2.15 > > Time Spent: 10m > Remaining Estimate: 0h > > There are several places where thin client threads can hang: > GridNioClientConnectionMultiplexer: SSL enabled + PA enabled > {noformat} > // Can return finished future with error -> sslHandshakeFut will not complete. > GridNioFuture sesFut = srv.createSession(ch, meta, false, > null); > if (sslHandshakeFut != null) > sslHandshakeFut.get(); > {noformat} > TcpClientChannel: PA enabled > {noformat} > ClientRequestFuture fut = new ClientRequestFuture(requestId, > ClientOperation.HANDSHAKE); > // TcpClientChannel may be closed and fut will not complete. > assert !closed.get(); > pendingReqs.put(requestId, fut); > ... > ByteBuffer buf = timeout > 0 ? fut.get(timeout) : fut.get(); > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18836) Incremental snapshot restore should fail in case of corrupted WAL segment
Maksim Timonin created IGNITE-18836: --- Summary: Incremental snapshot restore should fail in case of corrupted WAL segment Key: IGNITE-18836 URL: https://issues.apache.org/jira/browse/IGNITE-18836 Project: Ignite Issue Type: Improvement Reporter: Maksim Timonin There were cases when failure during parsing WAL segments are just skipped. It's required to investigate all possible cases, and add tests for that. IncSnapshot should fail if any WAL segment is corrupted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18814) Fix ItTxDistributedTestSingleNode#assertPartitionsSame
[ https://issues.apache.org/jira/browse/IGNITE-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladislav Pyatkov updated IGNITE-18814: --- Description: *Motivation* {code:java} MvPartitionStorage storage = listener.getMvStorage(); if (hash == 0) { hash = storage.hashCode(); } else if (hash != storage.hashCode()) { return false; }{code} This code is wrong, hash code is not defined for storage. *Implementation notes* To summarize all the mentioned before, the main issue is finding a way to compare data stored in different replicas. The obvious approach to calculate hash all object that stored in each replica storage. But the implementation can be difficult and time expensive in the test. The easy way to compare stored indexies ( _MvPartitionStorage#lastAppliedIndex_ , _MvPartitionStorage#persistedIndex_). The both are applicable for comparison, but _persistedIndex_ upgrades latter. {code} MvPartitionStorage storage = listener.getMvStorage(); if (idx == 0) { idx = storage.persistedIndex(); } else if (idx != storage.persistedIndex()) { return false; } {code} was: {code:java} MvPartitionStorage storage = listener.getMvStorage(); if (hash == 0) { hash = storage.hashCode(); } else if (hash != storage.hashCode()) { return false; }{code} This code is wrong, hash code is not defined for storage. To summarize all the mentioned before, the main issue is finding a way to compare data stored in different replicas. The obvious approach to calculate hash all object that stored in each replica storage. But the implementation can be difficult and time expensive in the test. The easy way to compare stored indexies ( _MvPartitionStorage#lastAppliedIndex_ , _MvPartitionStorage#persistedIndex_). The both are applicable for comparison, but _persistedIndex_ upgrades latter. {code} MvPartitionStorage storage = listener.getMvStorage(); if (idx == 0) { idx = storage.persistedIndex(); } else if (idx != storage.persistedIndex()) { return false; } {code} > Fix ItTxDistributedTestSingleNode#assertPartitionsSame > -- > > Key: IGNITE-18814 > URL: https://issues.apache.org/jira/browse/IGNITE-18814 > Project: Ignite > Issue Type: Bug >Reporter: Ivan Bessonov >Priority: Critical > Labels: ignite-3 > > *Motivation* > {code:java} > MvPartitionStorage storage = listener.getMvStorage(); > if (hash == 0) { > hash = storage.hashCode(); > } else if (hash != storage.hashCode()) { > return false; > }{code} > This code is wrong, hash code is not defined for storage. > *Implementation notes* > To summarize all the mentioned before, the main issue is finding a way to > compare data stored in different replicas. > The obvious approach to calculate hash all object that stored in each replica > storage. But the implementation can be difficult and time expensive in the > test. > The easy way to compare stored indexies ( > _MvPartitionStorage#lastAppliedIndex_ , _MvPartitionStorage#persistedIndex_). > The both are applicable for comparison, but _persistedIndex_ upgrades latter. > {code} > MvPartitionStorage storage = listener.getMvStorage(); > if (idx == 0) { > idx = storage.persistedIndex(); > } else if (idx != storage.persistedIndex()) { > return false; > } > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18814) Fix ItTxDistributedTestSingleNode#assertPartitionsSame
[ https://issues.apache.org/jira/browse/IGNITE-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladislav Pyatkov updated IGNITE-18814: --- Description: {code:java} MvPartitionStorage storage = listener.getMvStorage(); if (hash == 0) { hash = storage.hashCode(); } else if (hash != storage.hashCode()) { return false; }{code} This code is wrong, hash code is not defined for storage. To summarize all the mentioned before, the main issue is finding a way to compare data stored in different replicas. The obvious approach to calculate hash all object that stored in each replica storage. But the implementation can be difficult and time expensive in the test. The easy way to compare stored indexies ( _MvPartitionStorage#lastAppliedIndex_ , _MvPartitionStorage#persistedIndex_). The both are applicable for comparison, but _persistedIndex_ upgrades latter. {code} MvPartitionStorage storage = listener.getMvStorage(); if (idx == 0) { idx = storage.persistedIndex(); } else if (idx != storage.persistedIndex()) { return false; } {code} was: {code:java} MvPartitionStorage storage = listener.getMvStorage(); if (hash == 0) { hash = storage.hashCode(); } else if (hash != storage.hashCode()) { return false; }{code} This code is wrong, hash code is not defined for storages > Fix ItTxDistributedTestSingleNode#assertPartitionsSame > -- > > Key: IGNITE-18814 > URL: https://issues.apache.org/jira/browse/IGNITE-18814 > Project: Ignite > Issue Type: Bug >Reporter: Ivan Bessonov >Priority: Critical > Labels: ignite-3 > > {code:java} > MvPartitionStorage storage = listener.getMvStorage(); > if (hash == 0) { > hash = storage.hashCode(); > } else if (hash != storage.hashCode()) { > return false; > }{code} > This code is wrong, hash code is not defined for storage. > To summarize all the mentioned before, the main issue is finding a way to > compare data stored in different replicas. > The obvious approach to calculate hash all object that stored in each replica > storage. But the implementation can be difficult and time expensive in the > test. > The easy way to compare stored indexies ( > _MvPartitionStorage#lastAppliedIndex_ , _MvPartitionStorage#persistedIndex_). > The both are applicable for comparison, but _persistedIndex_ upgrades latter. > {code} > MvPartitionStorage storage = listener.getMvStorage(); > if (idx == 0) { > idx = storage.persistedIndex(); > } else if (idx != storage.persistedIndex()) { > return false; > } > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-13963) Excessive classloading of PlatformDotNetSessionLockResult on thin client connection
[ https://issues.apache.org/jira/browse/IGNITE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690403#comment-17690403 ] Aleksey Plekhanov commented on IGNITE-13963: [~antkr] are you going to continue work with this ticket? Looks like the patch is ready but you need to fix some comments. If it's not in your plans, can you unassign it? > Excessive classloading of PlatformDotNetSessionLockResult on thin client > connection > --- > > Key: IGNITE-13963 > URL: https://issues.apache.org/jira/browse/IGNITE-13963 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.9 >Reporter: Anton Kurbanov >Assignee: Anton Kurbanov >Priority: Major > Labels: ise > Time Spent: 0.5h > Remaining Estimate: 0h > > Empty constructor must be added to > org.apache.ignite.internal.processors.platform.websession.PlatformDotNetSessionLockResult. > Multiple thin clients connections are causing this class to generate > constructors: > {code:java} > [Loaded *sun.reflect.GeneratedSerializationConstructorAccessor99* from > __JVM_DefineClass__] > {code} > This is caused by U.forceEmptyConstructor(cls) call from > org.apache.ignite.internal.binary.BinaryClassDescriptor#constructor. > With lots of clients it is possible to create high load on metaspace which > results in lots of metaspace GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18835) Get rid of skipping safe time waiting on a primary node.
Mirza Aliev created IGNITE-18835: Summary: Get rid of skipping safe time waiting on a primary node. Key: IGNITE-18835 URL: https://issues.apache.org/jira/browse/IGNITE-18835 Project: Ignite Issue Type: Bug Reporter: Mirza Aliev Currently, we have a logic of skipping safe time waiting on a primary node in {{PartitionReplicaListener}} for read-only transaction, for example in {{PartitionReplicaListener#processReadOnlySingleEntryAction}} {code:java} CompletableFuture safeReadFuture = isPrimary ? completedFuture(null) : safeTime.waitFor(request.readTimestamp()); return safeReadFuture.thenCompose(unused -> resolveRowByPkForReadOnly(searchRow, readTimestamp)); {code} This is a wrong behaviour, we expect that this awaiting must be done on a primary node as well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18737) Tests from ItIgniteNodeRestartTest became flaky
[ https://issues.apache.org/jira/browse/IGNITE-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Uttsel updated IGNITE-18737: --- Description: h3. *Motivation* After https://issues.apache.org/jira/browse/IGNITE-18088 test from ItIgniteNodeRestartTest started to fail. Need to investigate the reason of fails {noformat} Caused by: java.util.concurrent.CancellationException at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:491) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:473) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.refreshAndGetLeaderWithTerm(RaftGroupServiceImpl.java:232) at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.ensureReplicaIsPrimary(PartitionReplicaListener.java:1880) at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.invoke(PartitionReplicaListener.java:275) at org.apache.ignite.internal.replicator.Replica.processRequest(Replica.java:61) at org.apache.ignite.internal.replicator.ReplicaManager.lambda$new$5(ReplicaManager.java:162) at org.apache.ignite.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:279) at org.apache.ignite.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:226) at org.apache.ignite.network.DefaultMessagingService.invoke(DefaultMessagingService.java:154) at org.apache.ignite.internal.replicator.ReplicaService.sendToReplica(ReplicaService.java:91) at org.apache.ignite.internal.replicator.ReplicaService.invoke(ReplicaService.java:178) at org.apache.ignite.internal.tx.impl.TxManagerImpl.cleanup(TxManagerImpl.java:150) at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processTxFinishAction$42(PartitionReplicaListener.java:984) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$38(RaftGroupServiceImpl.java:526) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183){noformat} The root cause is that distribution zones data nodes watch listener in TableManager invokes updatePendingAssignmentsKeys with the same data nodes as in table configuration assignments so RaftGroupServiceImpl stops and new the same RaftGroupServiceImpl starts. It cancels invokes to old RaftGroupServiceImpl so we have CancellationException at sendWithRetry. h3. *Implementation Notes* It is not need to change the assignments if the calculated assignments equals to the assignments in the table configuration. was: After https://issues.apache.org/jira/browse/IGNITE-18088 test from ItIgniteNodeRestartTest started to fail. Need to investigate the reason of fails https://ci.ignite.apache.org/test/-8438821239346742957?currentProjectId=ApacheIgnite3xGradle_Test_IntegrationTests=true=%3Cdefault%3E {noformat} org.apache.ignite.lang.IgniteInternalException: IGN-CMN-65535 TraceId:9b2c2e61-1676-4dc3-8d76-2494803d7a2b No such partition 0 in table TABLE1 at org.apache.ignite.internal.table.distributed.storage.InternalTableImpl.partitionRaftGroupService(InternalTableImpl.java:1124) at org.apache.ignite.internal.table.distributed.TableManager.lambda$createPartitionMover$79(TableManager.java:2131) at org.apache.ignite.internal.table.distributed.PartitionMover.movePartition(PartitionMover.java:66) at org.apache.ignite.internal.table.distributed.raft.RebalanceRaftGroupEventsListener.lambda$onLeaderElected$0(RebalanceRaftGroupEventsListener.java:202) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at
[jira] [Created] (IGNITE-18834) [ducktest] Measure WAL rebuild index performance
Nikolay Izhikov created IGNITE-18834: Summary: [ducktest] Measure WAL rebuild index performance Key: IGNITE-18834 URL: https://issues.apache.org/jira/browse/IGNITE-18834 Project: Ignite Issue Type: Task Reporter: Nikolay Izhikov We need ducktest to measure rebuild index.bin file performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18832) Option to assign specific name to thin client and capturing the name in sys.client_connections
[ https://issues.apache.org/jira/browse/IGNITE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Izhikov updated IGNITE-18832: - Labels: IEP-35 (was: ) > Option to assign specific name to thin client and capturing the name in > sys.client_connections > -- > > Key: IGNITE-18832 > URL: https://issues.apache.org/jira/browse/IGNITE-18832 > Project: Ignite > Issue Type: Improvement > Components: thin client >Affects Versions: 2.12, 2.13, 2.11.1, 2.14 >Reporter: Gangaiah >Priority: Major > Labels: IEP-35 > > > System view client_connections doesn't have name column to capture client > name, it should have the column to track the thin client connections.. It's > difficult to pin point the client where remote address same for multiple > clients in production scenarios. > https://lists.apache.org/thread/joy6vjopq2zovkj768hnhfvmd44vhs4b -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18763) Concurrent Meta Storage Watch processing
[ https://issues.apache.org/jira/browse/IGNITE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandr Polovtcev updated IGNITE-18763: - Description: Currently Meta Storage Watches follow the given contract: # All listeners are processed sequentially in the order that they have been registered; # All listeners are processed in the same thread; # All notifications are grouped by a particular revision. After all events for a particular revision get processed, all modified keys are persisted into the Vault. Some components may rely on this to ensure that dependent components can observe the most recent component state, because they registered their Meta Storage listeners after the component they depend on, therefore it is guaranteed that that component has already processed current Meta Storage revision. This approach can be improved in terms of performance by enabling concurrent Watch processing in multiple threads. In order to do that, we need to check and fix the following places: # If a component depends on another component's state, this state must be extracted into a Versioned Value; # If a component updates its state, relying on the fact, that this state is always updated in a single thread, concurrent primitives must be introduced to ensure thread safety; # Check if it is possible to persist updated keys in the Vault individually, thus making it possible not to wait for all Watches for a particular revision to complete before starting to process updates belonging to the next revision. was: Our first approach to design metastorage concentrated on hiding metastorage revision from other components has shown that it complicates code and creates unexpected dependencies. One example of such dependencies is that notification of configuration changes is delivered to components in the same order as they appear in configuration. If two components require specific order of notifications it enforces the same order of their configuration which is absolutely unclear for developer or end-user (if some user-facing configuration is involved). In order to fix these complications we want to expose metastorage revision in the form of VersionedValue. In new API keys in metastorage are exposed to client components in the form of VersionedValues with revision as their integral part. Components will use this additional information to coordinate between each other and if necessary to allow their dependants to wait for a particular version of configuration being applied. In this task we need to define a set of requirements for a new API, define how it should look and behave. > Concurrent Meta Storage Watch processing > > > Key: IGNITE-18763 > URL: https://issues.apache.org/jira/browse/IGNITE-18763 > Project: Ignite > Issue Type: Task > Components: persistence >Reporter: Sergey Chugunov >Assignee: Aleksandr Polovtcev >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > Currently Meta Storage Watches follow the given contract: > # All listeners are processed sequentially in the order that they have been > registered; > # All listeners are processed in the same thread; > # All notifications are grouped by a particular revision. After all events > for a particular revision get processed, all modified keys are persisted into > the Vault. > Some components may rely on this to ensure that dependent components can > observe the most recent component state, because they registered their Meta > Storage listeners after the component they depend on, therefore it is > guaranteed that that component has already processed current Meta Storage > revision. > This approach can be improved in terms of performance by enabling concurrent > Watch processing in multiple threads. In order to do that, we need to check > and fix the following places: > # If a component depends on another component's state, this state must be > extracted into a Versioned Value; > # If a component updates its state, relying on the fact, that this state is > always updated in a single thread, concurrent primitives must be introduced > to ensure thread safety; > # Check if it is possible to persist updated keys in the Vault individually, > thus making it possible not to wait for all Watches for a particular revision > to complete before starting to process updates belonging to the next > revision. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18431) Sql. BitSet is not supported.
[ https://issues.apache.org/jira/browse/IGNITE-18431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18431: -- Description: When a table has BitMask column, all queries fail with "BITMASK is not supported yet". https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L396 was: When a table has UUID and/or BitMask column, all queries fail with "UUID is not supported yet". https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L396 > Sql. BitSet is not supported. > - > > Key: IGNITE-18431 > URL: https://issues.apache.org/jira/browse/IGNITE-18431 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > When a table has BitMask column, all queries fail with "BITMASK is not > supported yet". > https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L396 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18431) Sql. BitSet is not supported.
[ https://issues.apache.org/jira/browse/IGNITE-18431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18431: -- Description: When a table has BitMask column, all queries fail with "BITMASK is not supported yet". https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L412 was: When a table has BitMask column, all queries fail with "BITMASK is not supported yet". https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L396 > Sql. BitSet is not supported. > - > > Key: IGNITE-18431 > URL: https://issues.apache.org/jira/browse/IGNITE-18431 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > When a table has BitMask column, all queries fail with "BITMASK is not > supported yet". > https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L412 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18431) Sql. BitSet is not supported.
[ https://issues.apache.org/jira/browse/IGNITE-18431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18431: -- Summary: Sql. BitSet is not supported. (was: UUID and BitSet are not supported in SQL) > Sql. BitSet is not supported. > - > > Key: IGNITE-18431 > URL: https://issues.apache.org/jira/browse/IGNITE-18431 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > When a table has UUID and/or BitMask column, all queries fail with "UUID is > not supported yet". > https://github.com/apache/ignite-3/blob/316545c3a15135a6441bbcfa6725f960152408a5/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/util/TypeUtils.java#L396 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18833) Fix node configuration file render method and write tests
Mikhail Pochatkin created IGNITE-18833: -- Summary: Fix node configuration file render method and write tests Key: IGNITE-18833 URL: https://issues.apache.org/jira/browse/IGNITE-18833 Project: Ignite Issue Type: Improvement Reporter: Mikhail Pochatkin org.apache.ignite.internal.configuration.storage.LocalFileConfigurationStorage render configuration file as JSON but should use HOCON format. Also, need to write more tests for this storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18832) Option to assign specific name to thin client and capturing the name in sys.client_connections
Gangaiah created IGNITE-18832: -- Summary: Option to assign specific name to thin client and capturing the name in sys.client_connections Key: IGNITE-18832 URL: https://issues.apache.org/jira/browse/IGNITE-18832 Project: Ignite Issue Type: Improvement Components: thin client Affects Versions: 2.14, 2.11.1, 2.13, 2.12 Reporter: Gangaiah System view client_connections doesn't have name column to capture client name, it should have the column to track the thin client connections.. It's difficult to pin point the client where remote address same for multiple clients in production scenarios. https://lists.apache.org/thread/joy6vjopq2zovkj768hnhfvmd44vhs4b -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16181) Calcite engine. Handle arithmetic overflows
[ https://issues.apache.org/jira/browse/IGNITE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690351#comment-17690351 ] Ilhom Ulmasov commented on IGNITE-16181: [~alexpl] could, you please check the PR? > Calcite engine. Handle arithmetic overflows > --- > > Key: IGNITE-16181 > URL: https://issues.apache.org/jira/browse/IGNITE-16181 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksey Plekhanov >Assignee: Ilhom Ulmasov >Priority: Major > Labels: calcite2-required, calcite3-required, ise > Time Spent: 10m > Remaining Estimate: 0h > > Currently, we silently ignore arithmetic overflows and return the wrong > result, for example: > {noformat} > SELECT 2147483647 + 2147483647{noformat} > Returns -2. But SQL standard says that exception should be thrown: > {noformat} > 5) If the most specific type of the result of an arithmetic operation is > exact numeric, then Case: > a) If the operator is not division and the mathematical result of the > operation is not exactly representable with the precision and scale of the > result data type, then an exception condition is raised: data exception — > numeric value out of range.{noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18763) Concurrent Meta Storage Watch processing
[ https://issues.apache.org/jira/browse/IGNITE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandr Polovtcev updated IGNITE-18763: - Summary: Concurrent Meta Storage Watch processing (was: Prepare a design document for implementing VersionedValue API in metastorage) > Concurrent Meta Storage Watch processing > > > Key: IGNITE-18763 > URL: https://issues.apache.org/jira/browse/IGNITE-18763 > Project: Ignite > Issue Type: Task > Components: persistence >Reporter: Sergey Chugunov >Assignee: Aleksandr Polovtcev >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > Our first approach to design metastorage concentrated on hiding metastorage > revision from other components has shown that it complicates code and creates > unexpected dependencies. > One example of such dependencies is that notification of configuration > changes is delivered to components in the same order as they appear in > configuration. If two components require specific order of notifications it > enforces the same order of their configuration which is absolutely unclear > for developer or end-user (if some user-facing configuration is involved). > In order to fix these complications we want to expose metastorage revision in > the form of VersionedValue. In new API keys in metastorage are exposed to > client components in the form of VersionedValues with revision as their > integral part. > Components will use this additional information to coordinate between each > other and if necessary to allow their dependants to wait for a particular > version of configuration being applied. > In this task we need to define a set of requirements for a new API, define > how it should look and behave. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16181) Calcite engine. Handle arithmetic overflows
[ https://issues.apache.org/jira/browse/IGNITE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690350#comment-17690350 ] Ignite TC Bot commented on IGNITE-16181: {panel:title=Branch: [pull/10541/head] Base: [master] : No blockers found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel} {panel:title=Branch: [pull/10541/head] Base: [master] : New Tests (2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1} {color:#8b}Calcite SQL{color} [[tests 2|https://ci2.ignite.apache.org/viewLog.html?buildId=7050938]] * {color:#013220}IgniteCalciteTestSuite: StdSqlOperatorsTest.testArithmeticOverflow - PASSED{color} * {color:#013220}IgniteCalciteTestSuite: TableDmlIntegrationTest.testInsertValueOverflow - PASSED{color} {panel} [TeamCity *-- Run :: All* Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7051020buildTypeId=IgniteTests24Java8_RunAll] > Calcite engine. Handle arithmetic overflows > --- > > Key: IGNITE-16181 > URL: https://issues.apache.org/jira/browse/IGNITE-16181 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksey Plekhanov >Assignee: Ilhom Ulmasov >Priority: Major > Labels: calcite2-required, calcite3-required, ise > Time Spent: 10m > Remaining Estimate: 0h > > Currently, we silently ignore arithmetic overflows and return the wrong > result, for example: > {noformat} > SELECT 2147483647 + 2147483647{noformat} > Returns -2. But SQL standard says that exception should be thrown: > {noformat} > 5) If the most specific type of the result of an arithmetic operation is > exact numeric, then Case: > a) If the operator is not division and the mathematical result of the > operation is not exactly representable with the precision and scale of the > result data type, then an exception condition is raised: data exception — > numeric value out of range.{noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18790) Add SSL support for the Ignite JDBC driver
[ https://issues.apache.org/jira/browse/IGNITE-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-18790: -- Fix Version/s: 3.0.0-beta2 > Add SSL support for the Ignite JDBC driver > -- > > Key: IGNITE-18790 > URL: https://issues.apache.org/jira/browse/IGNITE-18790 > Project: Ignite > Issue Type: Task > Components: jdbc >Reporter: Aleksandr >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 50m > Remaining Estimate: 0h > > This ticket should be based on the changes from IGNITE-18578. > Add JDBC-level properties for SSL support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Description: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts / will be able to call the necessary methods (? - this looks like a correct approach). was: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > Fix For: 3.0.0-beta2 > > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts / will be able to call the necessary methods (? - this looks > like a correct approach). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Fix Version/s: 3.0.0-beta2 > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > Fix For: 3.0.0-beta2 > > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts (? - this looks like a correct approach). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Affects Version/s: (was: 3.0.0-beta2) > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts (? - this looks like a correct approach). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Description: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). Either of this approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. was: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string and adds no additional casts. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). Either of this approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 3.0.0-beta2 >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts (? - this looks like a correct approach). > Either of this approaches can help resolving another issue with dynamic > parameters:
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Description: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). was: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). Either of the above approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 3.0.0-beta2 >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts (? - this looks like a correct approach). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Description: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). Either of the above approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. was: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). Either of this approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 3.0.0-beta2 >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts (? - this looks like a correct approach). > Either of the above approaches can help resolving another issue with dynamic > parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type >
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Description: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string and adds no additional casts. There several possible approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes . * Pass type information to the execution runtime so it would add the necessary casts (? - this looks like a correct approach). Either of this approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. was: At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string and adds no additional casts. There multiple approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes. * Pass type information to the execution runtime so it would add the necessary casts (*). Either of this approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 3.0.0-beta2 >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string and > adds no additional casts. > There several possible approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes . > * Pass type information to the execution runtime so it would add the > necessary casts (? - this looks like a correct approach). > Either of this approaches can help resolving another issue with dynamic > parameters:
[jira] [Updated] (IGNITE-17056) Design rebalance cancel mechanism
[ https://issues.apache.org/jira/browse/IGNITE-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-17056: - Reviewer: Denis Chudov > Design rebalance cancel mechanism > - > > Key: IGNITE-17056 > URL: https://issues.apache.org/jira/browse/IGNITE-17056 > Project: Ignite > Issue Type: Task >Reporter: Mirza Aliev >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > > There are cases when a current leader cannot perform rebalance on specified > set of nodes, for example, when some node from the raft group permanently > fails with RaftError#ECATCHUP. For such scenario retry mechanism is > implemented in IGNITE-16801, but we cannot retry rebalance intent infinitely, > so there should be implemented mechanism for canceling a rebalance. > Naive canceling could be implemented by removing pending key and replacing it > with planned key. But this approach has several crucial limitations and may > cause inconsistency in the current rebalance protocol, for example, when > there is a race between cancel and applying new assignment to the stable key > from the new leader. We can remove pending key right before applying new > assignment to the stable key, so we cannot resolve peers to ClusterIds, which > is made on a union of pending and stable keys. > Also there is a case, when we can lost planned rebalance: > # Current leader retries failed rebalance > # Current leader stops being leader for some reasons and sleeps > # New leader performs rebalance and calls > RebalanceRaftGroupEventsListener#onNewPeersConfigurationApplied > # At this moment old leader wakes up and cancels the current rebalance, so > it removes pending and writes to it planned key. > # At this moment we receive > RebalanceRaftGroupEventsListener#onNewPeersConfigurationApplied from the new > leader, see that planned is empty, so we just delete pending key, but this is > not correct to delete this key as far as the rebalance that is associated to > the removed key hasn't been performed yet. > Also we should consider separating scenarios for recoverable and > unrecoverable errors, because it might be useless to retry rebalance, if some > participating node fails with unrecoverable error. > Seems like we should properly think about introducing some failure handling > for such exceptional scenarios. > New node role from https://issues.apache.org/jira/browse/IGNITE-17252 primary > replica, can help us to resolve this issue in a simplier way and cancel > rebalance from the primary replica. > > As a result of this issue we must design correct algorithm for cancelling > hanged rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18831) Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime.
[ https://issues.apache.org/jira/browse/IGNITE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov updated IGNITE-18831: -- Summary: Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by the execution runtime. (was: Sql. Dynamic parameters. Pass inferred types of dynamic parameters to the execution runtime.) > Sql. Dynamic parameters. Inferred types of dynamic parameters are not used by > the execution runtime. > > > Key: IGNITE-18831 > URL: https://issues.apache.org/jira/browse/IGNITE-18831 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 3.0.0-beta2 >Reporter: Maksim Zhuravkov >Priority: Minor > Labels: calcite2-required, calcite3-required, ignite-3 > > At the moment the execution runtime is not aware of dynamic parameter types > inferred at the validation stage and when the validation stage completes and > those types are thrown away. During the execution, the runtime uses the > _java_class_ of a type parameter to perform an operation, which fails in the > following case: > A query has a dynamic parameter (string) but the validator, by using implicit > cast rules inferred, the actual type for that dynamic parameter to be some > another type that can be implicitly created from a string. > {code:java} > CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); > INSERT INTO UUIDS VALUES(1, ?); > {code} > * Execution runtime loses type information of the dynamic parameter and > assumes it to be a string. > * At the runtime toInternal call is going to fail, because it expects the > type of this parameter to be UUID but dynamic parameter contains a string and > adds no additional casts. > There multiple approaches to resolve this issue: > * Add dynamic cast to all dynamic parameters after validation completes. > * Pass type information to the execution runtime so it would add the > necessary casts (*). > Either of this approaches can help resolving another issue with dynamic > parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type > System. Reject plans with not matching dynamic parameters types during query > validation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18831) Sql. Dynamic parameters. Pass inferred types of dynamic parameters to the execution runtime.
Maksim Zhuravkov created IGNITE-18831: - Summary: Sql. Dynamic parameters. Pass inferred types of dynamic parameters to the execution runtime. Key: IGNITE-18831 URL: https://issues.apache.org/jira/browse/IGNITE-18831 Project: Ignite Issue Type: Improvement Components: sql Affects Versions: 3.0.0-beta2 Reporter: Maksim Zhuravkov At the moment the execution runtime is not aware of dynamic parameter types inferred at the validation stage and when the validation stage completes and those types are thrown away. During the execution, the runtime uses the _java_class_ of a type parameter to perform an operation, which fails in the following case: A query has a dynamic parameter (string) but the validator, by using implicit cast rules inferred, the actual type for that dynamic parameter to be some another type that can be implicitly created from a string. {code:java} CREATE TABLE UUIDS (id INTEGER PRIMARY KEY, uuid_key UUID); INSERT INTO UUIDS VALUES(1, ?); {code} * Execution runtime loses type information of the dynamic parameter and assumes it to be a string. * At the runtime toInternal call is going to fail, because it expects the type of this parameter to be UUID but dynamic parameter contains a string and adds no additional casts. There multiple approaches to resolve this issue: * Add dynamic cast to all dynamic parameters after validation completes. * Pass type information to the execution runtime so it would add the necessary casts (*). Either of this approaches can help resolving another issue with dynamic parameters: https://issues.apache.org/jira/browse/IGNITE-18645: Sql. Type System. Reject plans with not matching dynamic parameters types during query validation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18830) GridQueryProcessor logs unnecessary error message
Benjamin Garaude created IGNITE-18830: - Summary: GridQueryProcessor logs unnecessary error message Key: IGNITE-18830 URL: https://issues.apache.org/jira/browse/IGNITE-18830 Project: Ignite Issue Type: Bug Affects Versions: 2.14 Reporter: Benjamin Garaude GridQueryProcessor logs error message of type: {quote}07:24:47.571 [build-idx-runner-#123%default-grid%|#123%default-grid%] ERROR o.a.i.i.p.q.GridQueryProcessor - Failed to rebuild indexes for cache [name=...] {quote} But there is no underlying error. After a quick analysis, this log is emitted by org.apache.ignite.internal.processors.query.GridQueryProcessor.java:2640 If the err is null and log.isInfoEnabled() is false, (which is our case) then it misses the if at line 2637, and goes to the else at line 2640. This is likely a log issue in that class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17197) Change the default storage engine (for tables)
[ https://issues.apache.org/jira/browse/IGNITE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690269#comment-17690269 ] Kirill Tkalenko commented on IGNITE-17197: -- Looks good! > Change the default storage engine (for tables) > -- > > Key: IGNITE-17197 > URL: https://issues.apache.org/jira/browse/IGNITE-17197 > Project: Ignite > Issue Type: Task >Reporter: Sergey Uttsel >Assignee: Aleksandr Polovtcev >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently the default pagememory storage doesn't support MVCC. So it doesn't > fit to the transaction protocol. Only RocksDbTableStorage supports now. So > after RocksDbTableStorage was integrated in > https://issues.apache.org/jira/browse/IGNITE-16881 the > TablesConfigurationSchema#defaultDataStorage was set to "rocksdb" and some > tests were disabled. > Need to: > # implement MvTableStorage and MvPartitionStorage by other storages. > # set TablesConfigurationSchema#defaultDataStorage to "pagememory" or other > storage. > # enable tests marked by "IGNITE-17197" > # check other places marked by IGNITE-17197" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18819) Permissions for cluster activation/deactivation.
[ https://issues.apache.org/jira/browse/IGNITE-18819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-18819: -- Labels: ise (was: ) > Permissions for cluster activation/deactivation. > > > Key: IGNITE-18819 > URL: https://issues.apache.org/jira/browse/IGNITE-18819 > Project: Ignite > Issue Type: Improvement >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Major > Labels: ise > Time Spent: 10m > Remaining Estimate: 0h > > Cluster activation/deactivation actions should have permissions like > JOIN_AS_SERVER, ADMIN_SNAPSHOT. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18023) Implement GC API in page Memory based MV partition storages
[ https://issues.apache.org/jira/browse/IGNITE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18023: - Reviewer: Ivan Bessonov (was: Aleksandr Polovtcev) > Implement GC API in page Memory based MV partition storages > --- > > Key: IGNITE-18023 > URL: https://issues.apache.org/jira/browse/IGNITE-18023 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Please refer to Epic for details. > What I assume should happen: we probably need a hard synchronization on > RowIds. And by that I mean a striped lock for reading/writing from/into a > chain. > I see no other options, but if there are better ideas - go ahead. > All tests should pass, maybe more tests added if there are problems > discovered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18733) Mechanism to synchronize database schema changes between nodes
[ https://issues.apache.org/jira/browse/IGNITE-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690245#comment-17690245 ] Kirill Tkalenko commented on IGNITE-18733: -- Should fix fluky *org.apache.ignite.internal.sql.engine.ItDmlTest#pkConstraintConsistencyTest* > Mechanism to synchronize database schema changes between nodes > -- > > Key: IGNITE-18733 > URL: https://issues.apache.org/jira/browse/IGNITE-18733 > Project: Ignite > Issue Type: Epic > Components: persistence >Reporter: Sergey Chugunov >Priority: Major > Labels: ignite-3 > > Usability of Apache Ignite as a distributed database to a large extent > depends on a solid support of dynamic schema changes. > DDL operations like adding of deleting columns of a table or (more > importantly) adding or dropping indexes require proper syncronization between > nodes and integration with a number of other components. > Regardless of implementation details synchronization mechanism: > * should allow (most) user operations and schema change operations to > execute concurrently; > * should allow access to the data despite of the version of its schema; > * should shorten periods of time when certain types of operations are > suspended or work ineffectively (e.g. because of rebuilding of an index). > Schema synchronization needs integration with the following components (the > list may not be exhaustive): > * SQL engine - provide information about start and stop of operations, > notify about [un]availability of certain indexes; > * TX engine - e.g. if a transaction is started concurrently with DDL > operations; > * GC for MV storages - garbage collector needs to be aware of schema changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18023) Implement GC API in page Memory based MV partition storages
[ https://issues.apache.org/jira/browse/IGNITE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18023: - Reviewer: Aleksandr Polovtcev > Implement GC API in page Memory based MV partition storages > --- > > Key: IGNITE-18023 > URL: https://issues.apache.org/jira/browse/IGNITE-18023 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Please refer to Epic for details. > What I assume should happen: we probably need a hard synchronization on > RowIds. And by that I mean a striped lock for reading/writing from/into a > chain. > I see no other options, but if there are better ideas - go ahead. > All tests should pass, maybe more tests added if there are problems > discovered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18821) Add generated REST spec to doc
[ https://issues.apache.org/jira/browse/IGNITE-18821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Gusev reassigned IGNITE-18821: --- Assignee: Igor Gusev > Add generated REST spec to doc > -- > > Key: IGNITE-18821 > URL: https://issues.apache.org/jira/browse/IGNITE-18821 > Project: Ignite > Issue Type: Task >Reporter: Igor Gusev >Assignee: Igor Gusev >Priority: Major > Labels: ignite-3 > > Currently, we have to direct users to REST spec. We should add a rest spec > page to documentation to make it easier to read. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (IGNITE-18236) Cache objects transformation
[ https://issues.apache.org/jira/browse/IGNITE-18236 ] Anton Vinogradov deleted comment on IGNITE-18236: --- was (Author: ignitetcbot): {panel:title=Branch: [pull/10393/head] Base: [master] : No blockers found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel} {panel:title=Branch: [pull/10393/head] Base: [master] : New Tests (42)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1} {color:#8b}Index Query API{color} [[tests 8|https://ci2.ignite.apache.org/viewLog.html?buildId=7043354]] * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testValueAliasUpperField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testValueAliasField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testValueLowerField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testKeyAliasField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testValueField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testKeyLowerField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testKeyField - PASSED{color} * {color:#013220}IndexQueryTestSuite: IndexQueryCacheKeyValueTransformedFieldsTest.testKeyAliasUpperField - PASSED{color} {color:#8b}Cache 13{color} [[tests 12|https://ci2.ignite.apache.org/viewLog.html?buildId=7043316]] * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationTest.testTransformable[mode=ATOMIC] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationTest.testUntransformable[mode=ATOMIC] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationTest.testTransformable[mode=TRANSACTIONAL] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationTest.testUntransformable[mode=TRANSACTIONAL] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testString[mode=TRANSACTIONAL] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testObject[mode=TRANSACTIONAL] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testBinaryObject[mode=TRANSACTIONAL] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testInt[mode=TRANSACTIONAL] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testString[mode=ATOMIC] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testObject[mode=ATOMIC] - PASSED{color} * {color:#013220}IgniteCacheTestSuite13: CacheObjectsTransformationEvolutionTest.testBinaryObject[mode=ATOMIC] - PASSED{color} ... and 1 new tests {color:#8b}Cache Objects Compression{color} [[tests 22|https://ci2.ignite.apache.org/viewLog.html?buildId=7043418]] * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testWrappedStringBinaryArray[mode=THIN_CLIENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testStringArray[mode=NODE] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testString[mode=NODE] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testIncompressible[mode=NODE] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testString[mode=THIN_CLIENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testIncompressible[mode=THIN_CLIENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testWrappedString[mode=THIN_CLIENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testWrappedStringArray[mode=THIN_CLIENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testWrappedString[mode=PERSISTENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testWrappedStringArray[mode=PERSISTENT] - PASSED{color} * {color:#013220}IgniteCompressionTestSuite: CacheObjectsCompressionConsumptionTest.testWrappedStringBinaryArray[mode=PERSISTENT] - PASSED{color} ... and 11 new tests {panel} [TeamCity *-- Run :: All* Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7043419buildTypeId=IgniteTests24Java8_RunAll] > Cache objects transformation > > > Key: IGNITE-18236 > URL: