[jira] [Resolved] (IGNITE-19850) Fix metastorage unable to recover with multiple ms nodes
[ https://issues.apache.org/jira/browse/IGNITE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov resolved IGNITE-19850. - Resolution: Won't Fix > Fix metastorage unable to recover with multiple ms nodes > > > Key: IGNITE-19850 > URL: https://issues.apache.org/jira/browse/IGNITE-19850 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > On recovery, metastorage refreshes raft leader to access latest metastorage > revision. If there is no leader yet, or not all metastorage nodes are online, > recovery won't proceed. If 3 nodes are being started synchronously and all 3 > nodes are metastorage nodes, then they will fail to start. > The easiest solution would be to not start nodes synchronously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19996) ItNodeTest#testNewPeersConfigurationAppliedListener become flaky
[
https://issues.apache.org/jira/browse/IGNITE-19996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov reassigned IGNITE-19996:
---
Assignee: Semyon Danilov
> ItNodeTest#testNewPeersConfigurationAppliedListener become flaky
>
>
> Key: IGNITE-19996
> URL: https://issues.apache.org/jira/browse/IGNITE-19996
> Project: Ignite
> Issue Type: Bug
>Reporter: Mirza Aliev
>Assignee: Semyon Danilov
>Priority: Major
> Labels: ignite-3
>
> {{ItNodeTest#testNewPeersConfigurationAppliedListener}} started to fail time
> to time with
> {noformat}
> org.opentest4j.AssertionFailedError: expected: but was:
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
> at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
> at
> app//org.apache.ignite.raft.jraft.core.ItNodeTest.testNewPeersConfigurationAppliedListener(ItNodeTest.java:3038)
> {noformat}
> Brief investigation showed that logs are full of
> {noformat}
> 2023-07-18 12:50:39:922 +0400
> [DEBUG][int_tnpcal_5006-client-6][HandshakeHandler] Error when performing
> handshake
> org.apache.ignite.internal.network.handshake.ChannelAlreadyExistsException
> at
> org.apache.ignite.internal.network.recovery.RecoveryClientHandshakeManager.onHandshakeStartMessage(RecoveryClientHandshakeManager.java:235)
> at
> org.apache.ignite.internal.network.recovery.RecoveryClientHandshakeManager.onMessage(RecoveryClientHandshakeManager.java:151)
> at
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
> {noformat}
> and
> {noformat}
> 2023-07-18 12:50:38:896 +0400
> [DEBUG][int_tnpcal_5003-srv-worker-2][HandshakeHandler] Error when performing
> handshake
> java.io.IOException: Connection reset by peer
> {noformat}
> Seems that sometimes it is impossible to perform {{changePeers}} from the
> test because we see this errors right after we try to perform this action:
> {noformat}
> 2023-07-18 12:50:39:710 +0400 [ERROR][Test worker][AbstractClientService]
> Fail to connect int_tnpcal_5006, exception: java.net.ConnectException.
> 2023-07-18 12:50:39:710 +0400 [ERROR][Test worker][ReplicatorGroupImpl] Fail
> to check replicator connection to peer=int_tnpcal_5006,
> replicatorType=Follower.
> 2023-07-18 12:50:39:710 +0400 [ERROR][Test worker][NodeImpl] Node
> start the replicator failed,
> peer=int_tnpcal_5006.
> {noformat}
> and
> {noformat}
> 2023-07-18 12:50:39:921 +0400
> [INFO][int_tnpcal_5006-client-6][RecoveryClientHandshakeManager] Failed to
> acquire recovery descriptor during handshake, it is held by: [id: 0x33e12310,
> L:/127.0.0.1:5006 - R:/127.0.0.1:64991]
> {noformat}
> Seems that IGNITE-19903 has brought this problems.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-19850) Fix metastorage unable to recover with multiple ms nodes
[ https://issues.apache.org/jira/browse/IGNITE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19850: --- Assignee: Semyon Danilov > Fix metastorage unable to recover with multiple ms nodes > > > Key: IGNITE-19850 > URL: https://issues.apache.org/jira/browse/IGNITE-19850 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > On recovery, metastorage refreshes raft leader to access latest metastorage > revision. If there is no leader yet, or not all metastorage nodes are online, > recovery won't proceed. If 3 nodes are being started synchronously and all 3 > nodes are metastorage nodes, then they will fail to start. > The easiest solution would be to not start nodes synchronously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19913) Potential performance degradation in TableManager#createTableLocally
Semyon Danilov created IGNITE-19913:
---
Summary: Potential performance degradation in
TableManager#createTableLocally
Key: IGNITE-19913
URL: https://issues.apache.org/jira/browse/IGNITE-19913
Project: Ignite
Issue Type: Bug
Reporter: Semyon Danilov
In IGNITE-19778 a race condition was fixed in the createTableLocally method:
{code:java}
return allOf(createPartsFut, fireEvent(TableEvent.CREATE, new
TableEventParameters(causalityToken, tableId)));
{code}
Earlier, createPartsFut was not necessary to be completed. However this can
lead to a performance degradation. Need to rework
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-19903) Fix race condition in recovery protocol
[ https://issues.apache.org/jira/browse/IGNITE-19903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19903: --- Assignee: Semyon Danilov > Fix race condition in recovery protocol > --- > > Key: IGNITE-19903 > URL: https://issues.apache.org/jira/browse/IGNITE-19903 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_IntegrationTests_ModuleNetwork/7332374?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildProblemsSection=true&expandBuildChangesSection=true > RecoveryDescriptor#70: > assert req != null; > Seems like recovery descriptor is being used by two channels at the same time > because old channel is only closed then handshake (and recovery) is finished > (See NettyServer) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19903) Fix race condition in recovery protocol
[ https://issues.apache.org/jira/browse/IGNITE-19903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19903: Description: https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_IntegrationTests_ModuleNetwork/7332374?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildProblemsSection=true&expandBuildChangesSection=true RecoveryDescriptor#70: assert req != null; Seems like recovery descriptor is being used by two channels at the same time because old channel is only closed then handshake (and recovery) is finished (See NettyServer) was:https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_IntegrationTests_ModuleNetwork/7332374?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildProblemsSection=true&expandBuildChangesSection=true > Fix race condition in recovery protocol > --- > > Key: IGNITE-19903 > URL: https://issues.apache.org/jira/browse/IGNITE-19903 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_IntegrationTests_ModuleNetwork/7332374?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildProblemsSection=true&expandBuildChangesSection=true > RecoveryDescriptor#70: > assert req != null; > Seems like recovery descriptor is being used by two channels at the same time > because old channel is only closed then handshake (and recovery) is finished > (See NettyServer) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19903) Fix race condition in recovery protocol
Semyon Danilov created IGNITE-19903: --- Summary: Fix race condition in recovery protocol Key: IGNITE-19903 URL: https://issues.apache.org/jira/browse/IGNITE-19903 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_IntegrationTests_ModuleNetwork/7332374?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildProblemsSection=true&expandBuildChangesSection=true -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19778) Restore components state on metastorage recovery
[ https://issues.apache.org/jira/browse/IGNITE-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19778: --- Assignee: Semyon Danilov > Restore components state on metastorage recovery > > > Key: IGNITE-19778 > URL: https://issues.apache.org/jira/browse/IGNITE-19778 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Upon metastorage signalling > (https://issues.apache.org/jira/browse/IGNITE-19777) that certain revision > that was available when recovery started is achieved by the local metastorage > storage, components should restore their state using a "snapshot" of the data > with that specific revision. For example, TableManager might want to start > tables that it sees in that snapshot. Note that tables that are missing from > the data snapshot, must be destroyed as they are now obsolete. > So state must be restored and outdated remains of the previous state must be > cleared out. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19850) Fix metastorage unable to recover with multiple ms nodes
[ https://issues.apache.org/jira/browse/IGNITE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19850: Description: On recovery, metastorage refreshes raft leader to access latest metastorage revision. If there is no leader yet, or not all metastorage nodes are online, recovery won't proceed. If 3 nodes are being started synchronously and all 3 nodes are metastorage nodes, then they will fail to start. The easiest solution would be to not start nodes synchronously. > Fix metastorage unable to recover with multiple ms nodes > > > Key: IGNITE-19850 > URL: https://issues.apache.org/jira/browse/IGNITE-19850 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > On recovery, metastorage refreshes raft leader to access latest metastorage > revision. If there is no leader yet, or not all metastorage nodes are online, > recovery won't proceed. If 3 nodes are being started synchronously and all 3 > nodes are metastorage nodes, then they will fail to start. > The easiest solution would be to not start nodes synchronously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19828) Optimize RAFT log encoder/decoder
[ https://issues.apache.org/jira/browse/IGNITE-19828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737622#comment-17737622 ] Semyon Danilov commented on IGNITE-19828: - Looks good to me! > Optimize RAFT log encoder/decoder > - > > Key: IGNITE-19828 > URL: https://issues.apache.org/jira/browse/IGNITE-19828 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Assignee: Ivan Bessonov >Priority: Major > Labels: ignite-3 > Time Spent: 50m > Remaining Estimate: 0h > > We need to introduce varints (list in jraft encoders V2) and omit serializing > configuration for DATA entries (just like omitting data in CONFIGURATION > entries). > Current size of log entries is too big -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19850) Fix metastorage unable to recover with multiple ms nodes
Semyon Danilov created IGNITE-19850: --- Summary: Fix metastorage unable to recover with multiple ms nodes Key: IGNITE-19850 URL: https://issues.apache.org/jira/browse/IGNITE-19850 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19777) Perform local metastorage recovery
[ https://issues.apache.org/jira/browse/IGNITE-19777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19777: --- Assignee: Semyon Danilov > Perform local metastorage recovery > --- > > Key: IGNITE-19777 > URL: https://issues.apache.org/jira/browse/IGNITE-19777 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > Upon recovery, metastorage should request current cluster-wide metastorage > revision and wait until this revision is achieved. > After that it is safe to continue starting other components depending on the > metastorage and perform their own recovery based on the metastorage data at > said revision. After recovery is performed, watches must be installed with > the revision + 1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19778) Restore components state on metastorage recovery
[ https://issues.apache.org/jira/browse/IGNITE-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19778: Description: Upon metastorage signalling (https://issues.apache.org/jira/browse/IGNITE-19777) that certain revision that was available when recovery started is achieved by the local metastorage storage, components should restore their state using a "snapshot" of the data with that specific revision. For example, TableManager might want to start tables that it sees in that snapshot. Note that tables that are missing from the data snapshot, must be destroyed as they are now obsolete. So state must be restored and outdated remains of the previous state must be cleared out. > Restore components state on metastorage recovery > > > Key: IGNITE-19778 > URL: https://issues.apache.org/jira/browse/IGNITE-19778 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Upon metastorage signalling > (https://issues.apache.org/jira/browse/IGNITE-19777) that certain revision > that was available when recovery started is achieved by the local metastorage > storage, components should restore their state using a "snapshot" of the data > with that specific revision. For example, TableManager might want to start > tables that it sees in that snapshot. Note that tables that are missing from > the data snapshot, must be destroyed as they are now obsolete. > So state must be restored and outdated remains of the previous state must be > cleared out. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19777) Perform local metastorage recovery
[ https://issues.apache.org/jira/browse/IGNITE-19777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19777: Description: Upon recovery, metastorage should request current cluster-wide metastorage revision and wait until this revision is achieved. After that it is safe to continue starting other components depending on the metastorage and perform their own recovery based on the metastorage data at said revision. After recovery is performed, watches must be installed with the revision + 1. > Perform local metastorage recovery > --- > > Key: IGNITE-19777 > URL: https://issues.apache.org/jira/browse/IGNITE-19777 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Upon recovery, metastorage should request current cluster-wide metastorage > revision and wait until this revision is achieved. > After that it is safe to continue starting other components depending on the > metastorage and perform their own recovery based on the metastorage data at > said revision. After recovery is performed, watches must be installed with > the revision + 1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19776) Disable metastorage compaction cluster-wide while join is in progress
[ https://issues.apache.org/jira/browse/IGNITE-19776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19776: Description: MetaStorage compaction must be disabled locally on the recovering node, so that we don't have to handle CompactionException on start. Also, it would be beneficial to disable compaction cluster wide while a node is joining. So when a node initiates join via CMG, compaction should be disabled. After recovery is finished, compaction must be enabled again. > Disable metastorage compaction cluster-wide while join is in progress > - > > Key: IGNITE-19776 > URL: https://issues.apache.org/jira/browse/IGNITE-19776 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > MetaStorage compaction must be disabled locally on the recovering node, so > that we don't have to handle CompactionException on start. > Also, it would be beneficial to disable compaction cluster wide while a node > is joining. > So when a node initiates join via CMG, compaction should be disabled. After > recovery is finished, compaction must be enabled again. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19776) Disable metastorage compaction cluster-wide while join is in progress
[ https://issues.apache.org/jira/browse/IGNITE-19776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19776: Summary: Disable metastorage compaction cluster-wide while join is in progress (was: Disable metastorager compaction cluster-wide while join is in progress) > Disable metastorage compaction cluster-wide while join is in progress > - > > Key: IGNITE-19776 > URL: https://issues.apache.org/jira/browse/IGNITE-19776 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19777) Perform local metastorage recovery
Semyon Danilov created IGNITE-19777: --- Summary: Perform local metastorage recovery Key: IGNITE-19777 URL: https://issues.apache.org/jira/browse/IGNITE-19777 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19778) Restore components state on metastorage recovery
Semyon Danilov created IGNITE-19778: --- Summary: Restore components state on metastorage recovery Key: IGNITE-19778 URL: https://issues.apache.org/jira/browse/IGNITE-19778 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19776) Disable metastorager compaction cluster-wide while join is in progress
Semyon Danilov created IGNITE-19776: --- Summary: Disable metastorager compaction cluster-wide while join is in progress Key: IGNITE-19776 URL: https://issues.apache.org/jira/browse/IGNITE-19776 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19732) Cache IDs of configured tables
[ https://issues.apache.org/jira/browse/IGNITE-19732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17733477#comment-17733477 ] Semyon Danilov commented on IGNITE-19732: - Thank you for the contribution. The patch looks good to me, merged to the main branch > Cache IDs of configured tables > -- > > Key: IGNITE-19732 > URL: https://issues.apache.org/jira/browse/IGNITE-19732 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 40m > Remaining Estimate: 0h > > IGNITE-19665 demonstrates that checks for 'whether a table with the given ID > exists' take too much time as each of them parses whole tables configuration. > We should cache the configured table IDs in a consistent way: that is, if > this set has changed in the configuration, the cache should be invalidated > (and refilled with fresh data). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19739) Create partition storages only once
[ https://issues.apache.org/jira/browse/IGNITE-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19739: Summary: Create partition storages only once (was: Create partition and index storages only once) > Create partition storages only once > --- > > Key: IGNITE-19739 > URL: https://issues.apache.org/jira/browse/IGNITE-19739 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Right now storages are created using getOrCreate, should be only create -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IGNITE-19739) Create partition and index storages only once
[ https://issues.apache.org/jira/browse/IGNITE-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reopened IGNITE-19739: - > Create partition and index storages only once > - > > Key: IGNITE-19739 > URL: https://issues.apache.org/jira/browse/IGNITE-19739 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Right now storages are created using getOrCreate, should be only create -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-19739) Create partition and index storages only once
[ https://issues.apache.org/jira/browse/IGNITE-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov resolved IGNITE-19739. - Resolution: Fixed > Create partition and index storages only once > - > > Key: IGNITE-19739 > URL: https://issues.apache.org/jira/browse/IGNITE-19739 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Right now storages are created using getOrCreate, should be only create -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19739) Create partition and index storages only once
[ https://issues.apache.org/jira/browse/IGNITE-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19739: Description: Right now storages are created using getOrCreate, should be only create > Create partition and index storages only once > - > > Key: IGNITE-19739 > URL: https://issues.apache.org/jira/browse/IGNITE-19739 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Right now storages are created using getOrCreate, should be only create -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19739) Create partition and index storages only once
Semyon Danilov created IGNITE-19739: --- Summary: Create partition and index storages only once Key: IGNITE-19739 URL: https://issues.apache.org/jira/browse/IGNITE-19739 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19713) Start partition storages only for locally assigned partitions
Semyon Danilov created IGNITE-19713: --- Summary: Start partition storages only for locally assigned partitions Key: IGNITE-19713 URL: https://issues.apache.org/jira/browse/IGNITE-19713 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov IGNITE-19363 introduces PartitionSet which is populated with all partitions by default (not only assigned ones). Need to only set partitions that are assigned locally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19712) Handle rebalance wrt indexes
Semyon Danilov created IGNITE-19712: --- Summary: Handle rebalance wrt indexes Key: IGNITE-19712 URL: https://issues.apache.org/jira/browse/IGNITE-19712 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov After IGNITE-19363, index storages are no longer lazily instantiated. Need to listen to assignment changes and start new storages -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-19669) Support boxed long in direct marshaller
[ https://issues.apache.org/jira/browse/IGNITE-19669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov resolved IGNITE-19669. - Resolution: Fixed > Support boxed long in direct marshaller > --- > > Key: IGNITE-19669 > URL: https://issues.apache.org/jira/browse/IGNITE-19669 > Project: Ignite > Issue Type: Bug >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19669) Support boxed long in direct marshaller
Semyon Danilov created IGNITE-19669: --- Summary: Support boxed long in direct marshaller Key: IGNITE-19669 URL: https://issues.apache.org/jira/browse/IGNITE-19669 Project: Ignite Issue Type: Bug Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19363) Split start of indexes and start of partition raft group nodes
[ https://issues.apache.org/jira/browse/IGNITE-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19363: --- Assignee: Semyon Danilov > Split start of indexes and start of partition raft group nodes > -- > > Key: IGNITE-19363 > URL: https://issues.apache.org/jira/browse/IGNITE-19363 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Right now there is a cyclic dependency between raft group nodes recovery on > start and start of indexes. To start indexes all raft nodes should be > started. And raft nodes perform index rebuild on start. Index rebuild is a > blocking operation which waits for the table to appear. That can't happen > until all raft nodes started. As there's a smaller number of stripes in > disruptor, blocking one disruptor block the start of another raft node, so > table can't appear and index rebuild can't be finished. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
[
https://issues.apache.org/jira/browse/IGNITE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov resolved IGNITE-19271.
-
Fix Version/s: 3.0.0-beta2
Resolution: Fixed
Fixed by IGNITE-19532
> Persist revision-safeTime mapping in meta-storage
> -
>
> Key: IGNITE-19271
> URL: https://issues.apache.org/jira/browse/IGNITE-19271
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Assignee: Semyon Danilov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> IEP-98 states:
> {code:java}
> When creating a message M telling the cluster about a schema update
> activation moment, choose the message timestamp Tm (moving safeTime forward)
> equal to Now, but assign Tu (activation moment) contained in that M to be
> Tm+DD {code}
> This is hard to achieve.
> h3. Problem
> We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
> not straightforward. This is because we have too many actors:
> * There's a {_}client{_}, that chooses Tu, because it's the only actor that
> can affect message content.
> * There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses
> Tm.
> * There's everybody else, who expect a correspondence between Tu and Tm.
> First two actors are important, because they have independent clocks, but
> must coordinate the same event. This is impossible with described protocol.
> h3. Discussion
> Let's consider these two solutions:
> # Client generates Tm.
> # Meta-storage generates Tu.
> Option 1 is out of question, there must be only a single node at any given
> moment in time, that's responsible for the linear order of time in messages.
> What about option 2? Since meta-storage doesn't know anything about commands
> semantics, it can't really generate any data. So this solution doesn't work
> either.
> h3. Solution
> Combined solution could be the following:
> * Client sends DD as part of the command (this is not a constant, user _can_
> configure it, if they really feel like doing it)
> * Meta-storage generates {{Tm}}
> * Every node, upon receiving the update, calculates {{Tu}}
> This could work, if nodes would have never been restarted. There's one
> problem that needs to be solved: recovering the values of {{Tm}} from the
> (old) data upon node restart.
> This can be achieved by persisting safeTime along with revision as a part of
> metadata, that can be retrieved back through the meta-storage service API.
> In other words:
> 1. Client sends
> {code:java}
> schema.latest = 5
> schema.5.data = ...
> schema.5.dd = 30s{code}
> 2. Lease-holder adds meta-data to the command:
> {code:java}
> safeTime = 10:10
> {code}
> 3. Meta-storage listener writes the data:
> {code:java}
> revision = 33
> schema.latest = 5
> schema.5.data = ...
> schema.5.dd = 30s
> revision.33.safeTime = 10:10:00{code}
>
> How can you read {{{}Tu{}}}:
> * read "{{{}schema.5.dd"{}}};
> * read its revision, it's 33;
> * read a timestamp of revision 33 via specialized API;
> * add two values together.
> h3. Implications and restrictions
> There's a cleanup process in the meta-storage. It will eventually remove any
> "revision.x.safeTime" values, because corresponding revision became obsolete.
> But, we should somehow preserve timestamps of revisions that are used by
> schemas. Such behaviour can be achieved, if components can reserve a
> revision, and meta-storage can't compact it unless the reservation has been
> revoked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
[
https://issues.apache.org/jira/browse/IGNITE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov reassigned IGNITE-19271:
---
Assignee: Semyon Danilov
> Persist revision-safeTime mapping in meta-storage
> -
>
> Key: IGNITE-19271
> URL: https://issues.apache.org/jira/browse/IGNITE-19271
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Assignee: Semyon Danilov
>Priority: Major
> Labels: ignite-3
>
> IEP-98 states:
> {code:java}
> When creating a message M telling the cluster about a schema update
> activation moment, choose the message timestamp Tm (moving safeTime forward)
> equal to Now, but assign Tu (activation moment) contained in that M to be
> Tm+DD {code}
> This is hard to achieve.
> h3. Problem
> We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
> not straightforward. This is because we have too many actors:
> * There's a {_}client{_}, that chooses Tu, because it's the only actor that
> can affect message content.
> * There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses
> Tm.
> * There's everybody else, who expect a correspondence between Tu and Tm.
> First two actors are important, because they have independent clocks, but
> must coordinate the same event. This is impossible with described protocol.
> h3. Discussion
> Let's consider these two solutions:
> # Client generates Tm.
> # Meta-storage generates Tu.
> Option 1 is out of question, there must be only a single node at any given
> moment in time, that's responsible for the linear order of time in messages.
> What about option 2? Since meta-storage doesn't know anything about commands
> semantics, it can't really generate any data. So this solution doesn't work
> either.
> h3. Solution
> Combined solution could be the following:
> * Client sends DD as part of the command (this is not a constant, user _can_
> configure it, if they really feel like doing it)
> * Meta-storage generates {{Tm}}
> * Every node, upon receiving the update, calculates {{Tu}}
> This could work, if nodes would have never been restarted. There's one
> problem that needs to be solved: recovering the values of {{Tm}} from the
> (old) data upon node restart.
> This can be achieved by persisting safeTime along with revision as a part of
> metadata, that can be retrieved back through the meta-storage service API.
> In other words:
> 1. Client sends
> {code:java}
> schema.latest = 5
> schema.5.data = ...
> schema.5.dd = 30s{code}
> 2. Lease-holder adds meta-data to the command:
> {code:java}
> safeTime = 10:10
> {code}
> 3. Meta-storage listener writes the data:
> {code:java}
> revision = 33
> schema.latest = 5
> schema.5.data = ...
> schema.5.dd = 30s
> revision.33.safeTime = 10:10:00{code}
>
> How can you read {{{}Tu{}}}:
> * read "{{{}schema.5.dd"{}}};
> * read its revision, it's 33;
> * read a timestamp of revision 33 via specialized API;
> * add two values together.
> h3. Implications and restrictions
> There's a cleanup process in the meta-storage. It will eventually remove any
> "revision.x.safeTime" values, because corresponding revision became obsolete.
> But, we should somehow preserve timestamps of revisions that are used by
> schemas. Such behaviour can be achieved, if components can reserve a
> revision, and meta-storage can't compact it unless the reservation has been
> revoked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-19598) Internal table scan timeouts if node doesn't have partition assigned
Semyon Danilov created IGNITE-19598:
---
Summary: Internal table scan timeouts if node doesn't have
partition assigned
Key: IGNITE-19598
URL: https://issues.apache.org/jira/browse/IGNITE-19598
Project: Ignite
Issue Type: Bug
Reporter: Semyon Danilov
See ItTableScanTest#testMvScan with readOnly=true.
You will see that node is selected by assignments.
{code:java}
// Any node from assignments will do it.
ClusterNode node0 =
CLUSTER_NODES.get(0).clusterNodes().stream().filter(clusterNode -> {
return assignments.contains(clusterNode.name());
}).findFirst().orElseThrow();
{code}
In case if we select a node is not in assignments list, the operation will
timeout instead of returning error. It sends error that replica is not ready,
but replica will never be ready, because it shouldn't be started on this
particular node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-19547) Switch index IDs from UUID to int
[ https://issues.apache.org/jira/browse/IGNITE-19547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727109#comment-17727109 ] Semyon Danilov commented on IGNITE-19547: - The patch looks good to me. > Switch index IDs from UUID to int > - > > Key: IGNITE-19547 > URL: https://issues.apache.org/jira/browse/IGNITE-19547 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, as indices are stored as named list items in the Configuration, > they are internally (to the configuration) identified by UUIDs. The same > UUIDs are also used to identify indices in the whole system. > We need to change the latter: that is, in addition to the internal IDs > (needed only for configuration), we need to generate integer IDs and use them > in the rest of the system to identify indices. > For now, index IDs should be generated using same global counter that will be > used to generate table and zone IDs. > Internal IDs will remain until we switch from storing tables/indices/zones in > the Configuration to storing them in the Catalog (this is out of scope of > this task). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19532) Introduce happends before relation between local meta storage safe time publication and completion of corresponding meta storage listners
[
https://issues.apache.org/jira/browse/IGNITE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov reassigned IGNITE-19532:
---
Assignee: Semyon Danilov
> Introduce happends before relation between local meta storage safe time
> publication and completion of corresponding meta storage listners
> -
>
> Key: IGNITE-19532
> URL: https://issues.apache.org/jira/browse/IGNITE-19532
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Semyon Danilov
>Priority: Blocker
> Labels: ignite-3
>
> h3. Motivation
> Let's assume that we have a component named LeaseTracker that locally stores
> leases in a map though meta storage updates
> {code:java}
> public class LeaseTracker implements PlacementDriver {
> /** Leases cache. */
> private final Map leases;
> ...
> private class UpdateListener implements WatchListener {
> @Override
> public CompletableFuture onUpdate(WatchEvent event) {
> for (EntryEvent entry : event.entryEvents()) {
> ...
> Lease lease = fromBytes(msEntry.value());
> leases.put(grpId, lease);
> ...
> return completedFuture(null);
> }
> }
> ...{code}
> and we want to await lease in a meta storage safe time bounded way.
> {code:java}
> public CompletableFuture getPrimaryReplica(ReplicationGroupId
> replicationGroupId, HybridTimestamp timestamp) {
> ...
> return msManager.clusterTime().waitFor(timestamp).thenApply(() -> {
> ...
> Lease lease0 = leases.get(replicationGroupId);
> if (lease.getExpirationTime().after(timestamp)) {
> return lease0;
> } else {
> return null;
> }
> ...
>}
> } {code}
> Currently that won't work properly
> because
> {code:java}
> msManager.clusterTime().waitFor(timestamp) {code}
> will be completed before local leases will be populated with data through
> meta storage events.
> Thus it'll be nice to complete clusterTime().waitFor(timestamp) futute (and
> generally speaking publish safe time) after all corresponding ms listeners
> are completed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18518) Sorted index scan may return obsolete row versions
[ https://issues.apache.org/jira/browse/IGNITE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-18518: Reviewer: Aleksandr Polovtcev > Sorted index scan may return obsolete row versions > -- > > Key: IGNITE-18518 > URL: https://issues.apache.org/jira/browse/IGNITE-18518 > Project: Ignite > Issue Type: Bug >Reporter: Ivan Bessonov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 20m > Remaining Estimate: 0h > > According to the MV store design, indexes must be filtered before returning > data to user. For example, imagine that a row is deleted. This means that > there's a tombstone. > But, there's still an entry in indexes, for column values that the row used > to have. These should not be visible to end user. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18518) Sorted index scan may return obsolete row versions
[ https://issues.apache.org/jira/browse/IGNITE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-18518: --- Assignee: Semyon Danilov (was: Ivan Bessonov) > Sorted index scan may return obsolete row versions > -- > > Key: IGNITE-18518 > URL: https://issues.apache.org/jira/browse/IGNITE-18518 > Project: Ignite > Issue Type: Bug >Reporter: Ivan Bessonov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > According to the MV store design, indexes must be filtered before returning > data to user. For example, imagine that a row is deleted. This means that > there's a tombstone. > But, there's still an entry in indexes, for column values that the row used > to have. These should not be visible to end user. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-14734) Implement compaction functionality management for meta storage.
[
https://issues.apache.org/jira/browse/IGNITE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-14734:
Description:
At present {{SimpleInMemoryKeyValueStorage}} already has compaction
functionality but it is question: who and when should invoke {{compact}} method.
h3. Upd:
* It's still an open question: who and when should invoke {{compact}} method.
* Besides that, it's required to fix storage compaction - IGNITE-16444
* Seems that we, might reuse inner ms cursors meta in order to prevent
compaction of cursors over witch we are currently iterating.
* It's still however possible that revision-based get(), range(), and watch(),
invoke(), etc will throw CompactionException on corresponding initial calls.
h3. UPD 2:
For this ticket we decided to implement time-based compaction by creating a
timestamp (watermark)->revision mapping. Watermark provider will be implemented
in a separate ticket: https://issues.apache.org/jira/browse/IGNITE-19417
h3. UPD 3:
Time to Revision mapping mechanism
1. We can leverage the current implementation of the MetaStorage which has a
hackish feature of MetaStorage RAFT group’s leader handles the write command’s
HybridTimestamp from the node that initiates the write operation. Leader
adjusts its clock and sets the current time of the adjusted clock to the
command, so that the new adjusted time will be replicated to the followers and
learners. We can then use this time to update the time to revision mapping.
The mapping itself can be stored in a RocksDB column family.
2. The time to revision mapping can be sacked in favor of replacement of
revisions with timestamps. Timestamp is also an 8-byte long that monotonously
increases, so it seems like it can be used instead of auto-incremented
revisions.
3. As soon as we have MetaStorage based on the ReplicaService (indirect usage
of RAFT groups via the primary replica), we will be able to generate timestamps
for commands on the lease holder (and we should get rid of the hack mentioned
in the first point).
We can tie the compaction of the MetaStorage with the Garbage Collection of
partitions and use low watermark value in the compaction process. All the rules
that apply to garbage collection should be applied to the compaction, i.e. we
don’t remove the entry with a timestamp below MetaStorage’s LWM if it’s the
only entry for a given key.
A scheduled background task should be triggering the compaction. We can also be
triggering the compaction out of schedule every time the LWM is changed.
In addition it may be beneficial to perform a compaction on every insertion,
just like we do with garbage collection, however this is an optimisation and
should be benchmarked.
MetaStorage’s low watermark:
1. MS LWM cannot be higher than the Partition LWM.
2. MS LWM cannot be higher than any cursor's timestamp.
3. MS LWM cannot be higher than the timestamp of any schema version for which a
tuple exists that is serialized using this schema.
4. MS LWM is increased as soon as new LWM doesn't contradict all of the 3
points above.
See schema synchronization IEP:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization
was:
At present {{SimpleInMemoryKeyValueStorage}} already has compaction
functionality but it is question: who and when should invoke {{compact}} method.
h3. Upd:
* It's still an open question: who and when should invoke {{compact}} method.
* Besides that, it's required to fix storage compaction - IGNITE-16444
* Seems that we, might reuse inner ms cursors meta in order to prevent
compaction of cursors over witch we are currently iterating.
* It's still however possible that revision-based get(), range(), and watch(),
invoke(), etc will throw CompactionException on corresponding initial calls.
h3. UPD 2:
For this ticket we decided to implement time-based compaction by creating a
timestamp (watermark)->revision mapping. Watermark provider will be implemented
in a separate ticket: https://issues.apache.org/jira/browse/IGNITE-19417
> Implement compaction functionality management for meta storage.
> ---
>
> Key: IGNITE-14734
> URL: https://issues.apache.org/jira/browse/IGNITE-14734
> Project: Ignite
> Issue Type: Improvement
>Reporter: Andrey N. Gura
>Assignee: Semyon Danilov
>Priority: Major
> Labels: iep-61, ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> At present {{SimpleInMemoryKeyValueStorage}} already has compaction
> functionality but it is question: who and when should invoke {{compact}}
> method.
> h3. Upd:
> * It's still an open question: who and when should invoke {{compact}} method.
> * Besides that, it's required to fix storage com
[jira] [Updated] (IGNITE-19437) Add nullability checks to generated messages
[
https://issues.apache.org/jira/browse/IGNITE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-19437:
Description:
{{@Transferable}} messages should support nullability checks.
If a field is not marked as {{@Nullable}} (n.b. arrays should be marked as
{{int @Nullable []}}), then it shouldn't be possible to set null (or not set a
value) to that field
was:`@Transferable` messages should support nullability checks
> Add nullability checks to generated messages
>
>
> Key: IGNITE-19437
> URL: https://issues.apache.org/jira/browse/IGNITE-19437
> Project: Ignite
> Issue Type: Improvement
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {{@Transferable}} messages should support nullability checks.
> If a field is not marked as {{@Nullable}} (n.b. arrays should be marked as
> {{int @Nullable []}}), then it shouldn't be possible to set null (or not set
> a value) to that field
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19457) Review nullability of network messages' fields
[ https://issues.apache.org/jira/browse/IGNITE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19457: Description: PlacementDriverMessage: groupId ReplicaRequest: groupId SnapshotRequestMessage: id ReadWriteMultiRowReplicaRequest: commitPartitionId ReadWriteReplicaRequest: transactionId ReadWriteSingleRowReplicaRequest: commitPartitionId > Review nullability of network messages' fields > -- > > Key: IGNITE-19457 > URL: https://issues.apache.org/jira/browse/IGNITE-19457 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > PlacementDriverMessage: > groupId > ReplicaRequest: > groupId > SnapshotRequestMessage: > id > ReadWriteMultiRowReplicaRequest: > commitPartitionId > ReadWriteReplicaRequest: > transactionId > ReadWriteSingleRowReplicaRequest: > commitPartitionId -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19457) Review nullability of network messages' fields
Semyon Danilov created IGNITE-19457: --- Summary: Review nullability of network messages' fields Key: IGNITE-19457 URL: https://issues.apache.org/jira/browse/IGNITE-19457 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19437) Add nullability checks to generated messages
[ https://issues.apache.org/jira/browse/IGNITE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19437: --- Assignee: Semyon Danilov > Add nullability checks to generated messages > > > Key: IGNITE-19437 > URL: https://issues.apache.org/jira/browse/IGNITE-19437 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 20m > Remaining Estimate: 0h > > `@Transferable` messages should support nullability checks -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19437) Add nullability checks to generated messages
[ https://issues.apache.org/jira/browse/IGNITE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19437: Summary: Add nullability checks to generated messages (was: Add NotNull check to generated messages) > Add nullability checks to generated messages > > > Key: IGNITE-19437 > URL: https://issues.apache.org/jira/browse/IGNITE-19437 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 20m > Remaining Estimate: 0h > > `@Transferable` messages should support NotNull annotation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19437) Add nullability checks to generated messages
[ https://issues.apache.org/jira/browse/IGNITE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19437: Description: `@Transferable` messages should support nullability checks (was: `@Transferable` messages should support NotNull annotation) > Add nullability checks to generated messages > > > Key: IGNITE-19437 > URL: https://issues.apache.org/jira/browse/IGNITE-19437 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 20m > Remaining Estimate: 0h > > `@Transferable` messages should support nullability checks -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19437) Add NotNull check to generated messages
[ https://issues.apache.org/jira/browse/IGNITE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19437: Description: `@Transferable` messages should support NotNull annotation > Add NotNull check to generated messages > --- > > Key: IGNITE-19437 > URL: https://issues.apache.org/jira/browse/IGNITE-19437 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > `@Transferable` messages should support NotNull annotation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19437) Add NotNull check to generated messages
Semyon Danilov created IGNITE-19437: --- Summary: Add NotNull check to generated messages Key: IGNITE-19437 URL: https://issues.apache.org/jira/browse/IGNITE-19437 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19417) Provide low watermark for metastorage compaction
[ https://issues.apache.org/jira/browse/IGNITE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19417: Description: We should implement a process of increasing a low watermark for the metastorage compaction. I believe it should be a scheduled task in the top component (IgniteImpl) because it will needed information about open cursors and partitions. MetaStorage compaction low watermark should be a subject to the following rules: 1. It cannot be higher than the Partition LWM. 2. It cannot be higher than any cursor's timestamp. 3. It cannot be higher than the timestamp of any schema version for which a tuple exists that is serialized using this schema. 4. It may be increased as soon as new LWM doesn't contradict all of the 3 points above. was: MetaStorage compaction low watermark should be a subject to the following rules: 1. It cannot be higher than the Partition LWM. 2. It cannot be higher than any cursor's timestamp. 3. It cannot be higher than the timestamp of any schema version for which a tuple exists that is serialized using this schema. 4. It may be increased as soon as new LWM doesn't contradict all of the 3 points above. > Provide low watermark for metastorage compaction > > > Key: IGNITE-19417 > URL: https://issues.apache.org/jira/browse/IGNITE-19417 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > We should implement a process of increasing a low watermark for the > metastorage compaction. I believe it should be a scheduled task in the top > component (IgniteImpl) because it will needed information about open cursors > and partitions. > MetaStorage compaction low watermark should be a subject to the following > rules: > 1. It cannot be higher than the Partition LWM. > 2. It cannot be higher than any cursor's timestamp. > 3. It cannot be higher than the timestamp of any schema version for which a > tuple exists that is serialized using this schema. > 4. It may be increased as soon as new LWM doesn't contradict all of the 3 > points above. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19417) Provide low watermark for metastorage compaction
[ https://issues.apache.org/jira/browse/IGNITE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19417: Description: MetaStorage compaction low watermark should be a subject to the following rules: 1. It cannot be higher than the Partition LWM. 2. It cannot be higher than any cursor's timestamp. 3. It cannot be higher than the timestamp of any schema version for which a tuple exists that is serialized using this schema. 4. It may be increased as soon as new LWM doesn't contradict all of the 3 points above. > Provide low watermark for metastorage compaction > > > Key: IGNITE-19417 > URL: https://issues.apache.org/jira/browse/IGNITE-19417 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > MetaStorage compaction low watermark should be a subject to the following > rules: > 1. It cannot be higher than the Partition LWM. > 2. It cannot be higher than any cursor's timestamp. > 3. It cannot be higher than the timestamp of any schema version for which a > tuple exists that is serialized using this schema. > 4. It may be increased as soon as new LWM doesn't contradict all of the 3 > points above. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-14734) Implement compaction functionality management for meta storage.
[
https://issues.apache.org/jira/browse/IGNITE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-14734:
Description:
At present {{SimpleInMemoryKeyValueStorage}} already has compaction
functionality but it is question: who and when should invoke {{compact}} method.
h3. Upd:
* It's still an open question: who and when should invoke {{compact}} method.
* Besides that, it's required to fix storage compaction - IGNITE-16444
* Seems that we, might reuse inner ms cursors meta in order to prevent
compaction of cursors over witch we are currently iterating.
* It's still however possible that revision-based get(), range(), and watch(),
invoke(), etc will throw CompactionException on corresponding initial calls.
h3. UPD 2:
For this ticket we decided to implement time-based compaction by creating a
timestamp (watermark)->revision mapping. Watermark provider will be implemented
in a separate ticket: https://issues.apache.org/jira/browse/IGNITE-19417
was:
At present {{SimpleInMemoryKeyValueStorage}} already has compaction
functionality but it is question: who and when should invoke {{compact}} method.
h3. Upd:
* It's still an open question: who and when should invoke {{compact}} method.
* Besides that, it's required to fix storage compaction - IGNITE-16444
* Seems that we, might reuse inner ms cursors meta in order to prevent
compaction of cursors over witch we are currently iterating.
* It's still however possible that revision-based get(), range(), and watch(),
invoke(), etc will throw CompactionException on corresponding initial calls.
UPD 2:
For this ticket we decided to implement time-based compaction by creating a
timestamp (watermark)->revision mapping. Watermark provider will be implemented
in a separate ticket: https://issues.apache.org/jira/browse/IGNITE-19417
> Implement compaction functionality management for meta storage.
> ---
>
> Key: IGNITE-14734
> URL: https://issues.apache.org/jira/browse/IGNITE-14734
> Project: Ignite
> Issue Type: Improvement
>Reporter: Andrey N. Gura
>Assignee: Semyon Danilov
>Priority: Major
> Labels: iep-61, ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At present {{SimpleInMemoryKeyValueStorage}} already has compaction
> functionality but it is question: who and when should invoke {{compact}}
> method.
> h3. Upd:
> * It's still an open question: who and when should invoke {{compact}} method.
> * Besides that, it's required to fix storage compaction - IGNITE-16444
> * Seems that we, might reuse inner ms cursors meta in order to prevent
> compaction of cursors over witch we are currently iterating.
> * It's still however possible that revision-based get(), range(), and
> watch(), invoke(), etc will throw CompactionException on corresponding
> initial calls.
> h3. UPD 2:
> For this ticket we decided to implement time-based compaction by creating a
> timestamp (watermark)->revision mapping. Watermark provider will be
> implemented in a separate ticket:
> https://issues.apache.org/jira/browse/IGNITE-19417
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-14734) Implement compaction functionality management for meta storage.
[
https://issues.apache.org/jira/browse/IGNITE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-14734:
Description:
At present {{SimpleInMemoryKeyValueStorage}} already has compaction
functionality but it is question: who and when should invoke {{compact}} method.
h3. Upd:
* It's still an open question: who and when should invoke {{compact}} method.
* Besides that, it's required to fix storage compaction - IGNITE-16444
* Seems that we, might reuse inner ms cursors meta in order to prevent
compaction of cursors over witch we are currently iterating.
* It's still however possible that revision-based get(), range(), and watch(),
invoke(), etc will throw CompactionException on corresponding initial calls.
UPD 2:
For this ticket we decided to implement time-based compaction by creating a
timestamp (watermark)->revision mapping. Watermark provider will be implemented
in a separate ticket: https://issues.apache.org/jira/browse/IGNITE-19417
was:
At present {{SimpleInMemoryKeyValueStorage}} already has compaction
functionality but it is question: who and when should invoke {{compact}} method.
h3. Upd:
* It's still an open question: who and when should invoke {{compact}} method.
* Besides that, it's required to fix storage compaction - IGNITE-16444
* Seems that we, might reuse inner ms cursors meta in order to prevent
compaction of cursors over witch we are currently iterating.
* It's still however possible that revision-based get(), range(), and watch(),
invoke(), etc will throw CompactionException on corresponding initial calls.
> Implement compaction functionality management for meta storage.
> ---
>
> Key: IGNITE-14734
> URL: https://issues.apache.org/jira/browse/IGNITE-14734
> Project: Ignite
> Issue Type: Improvement
>Reporter: Andrey N. Gura
>Assignee: Semyon Danilov
>Priority: Major
> Labels: iep-61, ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At present {{SimpleInMemoryKeyValueStorage}} already has compaction
> functionality but it is question: who and when should invoke {{compact}}
> method.
> h3. Upd:
> * It's still an open question: who and when should invoke {{compact}} method.
> * Besides that, it's required to fix storage compaction - IGNITE-16444
> * Seems that we, might reuse inner ms cursors meta in order to prevent
> compaction of cursors over witch we are currently iterating.
> * It's still however possible that revision-based get(), range(), and
> watch(), invoke(), etc will throw CompactionException on corresponding
> initial calls.
> UPD 2:
> For this ticket we decided to implement time-based compaction by creating a
> timestamp (watermark)->revision mapping. Watermark provider will be
> implemented in a separate ticket:
> https://issues.apache.org/jira/browse/IGNITE-19417
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-14734) Implement compaction functionality management for meta storage.
[
https://issues.apache.org/jira/browse/IGNITE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719237#comment-17719237
]
Semyon Danilov commented on IGNITE-14734:
-
For this ticket I decided to implement time-based compaction by creating a
timestamp (watermark)->revision mapping. Watermark provider will be implemented
in a separate ticket: https://issues.apache.org/jira/browse/IGNITE-19417
> Implement compaction functionality management for meta storage.
> ---
>
> Key: IGNITE-14734
> URL: https://issues.apache.org/jira/browse/IGNITE-14734
> Project: Ignite
> Issue Type: Improvement
>Reporter: Andrey N. Gura
>Assignee: Semyon Danilov
>Priority: Major
> Labels: iep-61, ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At present {{SimpleInMemoryKeyValueStorage}} already has compaction
> functionality but it is question: who and when should invoke {{compact}}
> method.
> h3. Upd:
> * It's still an open question: who and when should invoke {{compact}} method.
> * Besides that, it's required to fix storage compaction - IGNITE-16444
> * Seems that we, might reuse inner ms cursors meta in order to prevent
> compaction of cursors over witch we are currently iterating.
> * It's still however possible that revision-based get(), range(), and
> watch(), invoke(), etc will throw CompactionException on corresponding
> initial calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-19417) Provide low watermark for metastorage compaction
Semyon Danilov created IGNITE-19417: --- Summary: Provide low watermark for metastorage compaction Key: IGNITE-19417 URL: https://issues.apache.org/jira/browse/IGNITE-19417 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-14734) Implement compaction functionality management for meta storage.
[
https://issues.apache.org/jira/browse/IGNITE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov reassigned IGNITE-14734:
---
Assignee: Semyon Danilov (was: Andrey N. Gura)
> Implement compaction functionality management for meta storage.
> ---
>
> Key: IGNITE-14734
> URL: https://issues.apache.org/jira/browse/IGNITE-14734
> Project: Ignite
> Issue Type: Improvement
>Reporter: Andrey N. Gura
>Assignee: Semyon Danilov
>Priority: Major
> Labels: iep-61, ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At present {{SimpleInMemoryKeyValueStorage}} already has compaction
> functionality but it is question: who and when should invoke {{compact}}
> method.
> h3. Upd:
> * It's still an open question: who and when should invoke {{compact}} method.
> * Besides that, it's required to fix storage compaction - IGNITE-16444
> * Seems that we, might reuse inner ms cursors meta in order to prevent
> compaction of cursors over witch we are currently iterating.
> * It's still however possible that revision-based get(), range(), and
> watch(), invoke(), etc will throw CompactionException on corresponding
> initial calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19376) Replica-based MetaStorage
[ https://issues.apache.org/jira/browse/IGNITE-19376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19376: Description: Right now any node can submit a command to the MetaStorage raft group. This approach must be changed to the approach we have with partitions: replicas with a primary replica that actually submits commands to the (possibly non-colocated) raft group leader. > Replica-based MetaStorage > - > > Key: IGNITE-19376 > URL: https://issues.apache.org/jira/browse/IGNITE-19376 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > > Right now any node can submit a command to the MetaStorage raft group. This > approach must be changed to the approach we have with partitions: replicas > with a primary replica that actually submits commands to the (possibly > non-colocated) raft group leader. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19377) Remove conditional MetaStorage raft commands
Semyon Danilov created IGNITE-19377: --- Summary: Remove conditional MetaStorage raft commands Key: IGNITE-19377 URL: https://issues.apache.org/jira/browse/IGNITE-19377 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov When we have replica-based MetaStorage, we should not replicate conditional commands like IfThenElse. Replica’s leader should decide on the command’s output and replicate simple Insert commands. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19376) Replica-based MetaStorage
Semyon Danilov created IGNITE-19376: --- Summary: Replica-based MetaStorage Key: IGNITE-19376 URL: https://issues.apache.org/jira/browse/IGNITE-19376 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19363) Split start of indexes and start of partition raft group nodes
[ https://issues.apache.org/jira/browse/IGNITE-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19363: Description: Right now there is a cyclic dependency between raft group nodes recovery on start and start of indexes. To start indexes all raft nodes should be started. And raft nodes perform index rebuild on start. Index rebuild is a blocking operation which waits for the table to appear. That can't happen until all raft nodes started. As there's a smaller number of stripes in disruptor, blocking one disruptor block the start of another raft node, so table can't appear and index rebuild can't be finished. > Split start of indexes and start of partition raft group nodes > -- > > Key: IGNITE-19363 > URL: https://issues.apache.org/jira/browse/IGNITE-19363 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Right now there is a cyclic dependency between raft group nodes recovery on > start and start of indexes. To start indexes all raft nodes should be > started. And raft nodes perform index rebuild on start. Index rebuild is a > blocking operation which waits for the table to appear. That can't happen > until all raft nodes started. As there's a smaller number of stripes in > disruptor, blocking one disruptor block the start of another raft node, so > table can't appear and index rebuild can't be finished. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19362) Deadlock while restarting with existing raft snapshot and scheduled index rebuild
[ https://issues.apache.org/jira/browse/IGNITE-19362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19362: Description: Node can't restart if there's a raft snapshot of the partition and there is a need to rebuild index. This happens due to index rebuild depending on raft nodes start. All raft nodes in turn can't start because nodes that were started first block disruptors with waitForIndex call. > Deadlock while restarting with existing raft snapshot and scheduled index > rebuild > - > > Key: IGNITE-19362 > URL: https://issues.apache.org/jira/browse/IGNITE-19362 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Node can't restart if there's a raft snapshot of the partition and there is a > need to rebuild index. This happens due to index rebuild depending on raft > nodes start. All raft nodes in turn can't start because nodes that were > started first block disruptors with waitForIndex call. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19362) Deadlock while restarting with existing raft snapshot and scheduled index rebuild
[ https://issues.apache.org/jira/browse/IGNITE-19362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19362: Description: Node can't restart if there's a raft snapshot of the partition and there is a need to rebuild index. This happens due to index rebuild depending on raft nodes start. All raft nodes in turn can't start because nodes that were started first block disruptors with waitForIndex call. As a workaround, we can skip waiting for snapshot on partition raft node start (was: Node can't restart if there's a raft snapshot of the partition and there is a need to rebuild index. This happens due to index rebuild depending on raft nodes start. All raft nodes in turn can't start because nodes that were started first block disruptors with waitForIndex call.) > Deadlock while restarting with existing raft snapshot and scheduled index > rebuild > - > > Key: IGNITE-19362 > URL: https://issues.apache.org/jira/browse/IGNITE-19362 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Node can't restart if there's a raft snapshot of the partition and there is a > need to rebuild index. This happens due to index rebuild depending on raft > nodes start. All raft nodes in turn can't start because nodes that were > started first block disruptors with waitForIndex call. As a workaround, we > can skip waiting for snapshot on partition raft node start -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19363) Split start of indexes and start of partition raft group nodes
Semyon Danilov created IGNITE-19363: --- Summary: Split start of indexes and start of partition raft group nodes Key: IGNITE-19363 URL: https://issues.apache.org/jira/browse/IGNITE-19363 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19362) Deadlock while restarting with existing raft snapshot and scheduled index rebuild
Semyon Danilov created IGNITE-19362: --- Summary: Deadlock while restarting with existing raft snapshot and scheduled index rebuild Key: IGNITE-19362 URL: https://issues.apache.org/jira/browse/IGNITE-19362 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov Assignee: Semyon Danilov Fix For: 3.0.0-beta2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19327) Fix a bug on scheduling a new low watermark update
[
https://issues.apache.org/jira/browse/IGNITE-19327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17714536#comment-17714536
]
Semyon Danilov commented on IGNITE-19327:
-
This patch looks good to me
> Fix a bug on scheduling a new low watermark update
> --
>
> Key: IGNITE-19327
> URL: https://issues.apache.org/jira/browse/IGNITE-19327
> Project: Ignite
> Issue Type: Bug
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I found an error in the logs, I need to fix it.
> {noformat}
> 2023-04-20 11:19:35:279 +0300
> [ERROR][%sqllogic1%low-watermark-updater-0][LowWatermark] Failed to update
> low watermark, will schedule again: HybridTimestamp [physical=1681976065287,
> logical=0]
> java.util.concurrent.CompletionException: java.lang.AssertionError: previous
> scheduled task has not finished
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at
> java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
> at
> java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:799)
> at
> java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2121)
> at
> org.apache.ignite.internal.table.distributed.LowWatermark.lambda$updateLowWatermark$10(LowWatermark.java:186)
> at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:889)
> at
> org.apache.ignite.internal.table.distributed.LowWatermark.updateLowWatermark(LowWatermark.java:175)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.AssertionError: previous scheduled task has not finished
> at
> org.apache.ignite.internal.table.distributed.LowWatermark.scheduleUpdateLowWatermarkBusy(LowWatermark.java:214)
> at
> org.apache.ignite.internal.table.distributed.LowWatermark.runGcAndScheduleUpdateLowWatermarkBusy(LowWatermark.java:208)
> at
> org.apache.ignite.internal.table.distributed.LowWatermark.lambda$updateLowWatermark$7(LowWatermark.java:189)
> at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:889)
> at
> org.apache.ignite.internal.table.distributed.LowWatermark.lambda$updateLowWatermark$8(LowWatermark.java:186)
> at
> java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:815)
> ... 12 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19209) Implement installing table schema updates
[ https://issues.apache.org/jira/browse/IGNITE-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19209: Description: At the moment there is a CatalogService that manages SchemaDescriptors (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is being handled by the CatalogService, however it seems like it doesn't actually write the schema to the metastorage. I believe that two things must be done in the scope of this ticket: 1. Actually write new schema to the metastorage. 2. Use node's HLC + DD (Delay Duration) which is described in the https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization as the new schema's activation time. was: At the moment there is a CatalogService that manages SchemaDescriptors (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is being handled by the CatalogService, however it seems like it doesn't actually write the schema to the metastorage. I believe that two things must be done in the scope of this ticket: 1. Actually write new schema to the metastorage. 2. Use node's HLC + DD (Delay Duration which is described in the https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization) as the new schema's activation time. > Implement installing table schema updates > - > > Key: IGNITE-19209 > URL: https://issues.apache.org/jira/browse/IGNITE-19209 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > At the moment there is a CatalogService that manages SchemaDescriptors > (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is > being handled by the CatalogService, however it seems like it doesn't > actually write the schema to the metastorage. > I believe that two things must be done in the scope of this ticket: > 1. Actually write new schema to the metastorage. > 2. Use node's HLC + DD (Delay Duration) which is described in the > https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization > as the new schema's activation time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19209) Implement installing table schema updates
[ https://issues.apache.org/jira/browse/IGNITE-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19209: Description: At the moment there is a CatalogService that manages SchemaDescriptors (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is being handled by the CatalogService, however it seems like it doesn't actually write the schema to the metastorage. I believe that two things must be done in the scope of this ticket: 1. Actually write new schema to the metastorage. 2. Use node's HLC + DD (Delay Duration which is described in the https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization) as the new schema's activation time. was: At the moment there is a CatalogService that manages SchemaDescriptors (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is being handled by the CatalogService, however it seems like it doesn't actually write the schema to the metastorage. I believe that two things must be done in the scope of this ticket: 1. Actually write new schema to the metastorage. 2. Use metastorage's cluster time as the new schema's activation time. We must also make use of Delay Duration which is described in the https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization > Implement installing table schema updates > - > > Key: IGNITE-19209 > URL: https://issues.apache.org/jira/browse/IGNITE-19209 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > At the moment there is a CatalogService that manages SchemaDescriptors > (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is > being handled by the CatalogService, however it seems like it doesn't > actually write the schema to the metastorage. > I believe that two things must be done in the scope of this ticket: > 1. Actually write new schema to the metastorage. > 2. Use node's HLC + DD (Delay Duration which is described in the > https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization) > as the new schema's activation time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19209) Implement installing table schema updates
[ https://issues.apache.org/jira/browse/IGNITE-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19209: Description: At the moment there is a CatalogService that manages SchemaDescriptors (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is being handled by the CatalogService, however it seems like it doesn't actually write the schema to the metastorage. I believe that two things must be done in the scope of this ticket: 1. Actually write new schema to the metastorage. 2. Use metastorage's cluster time as the new schema's activation time. We must also make use of Delay Duration which is described in the https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization was:TBD > Implement installing table schema updates > - > > Key: IGNITE-19209 > URL: https://issues.apache.org/jira/browse/IGNITE-19209 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > At the moment there is a CatalogService that manages SchemaDescriptors > (database schemas, not table schemas) and TableDescriptors. CREATE TABLE is > being handled by the CatalogService, however it seems like it doesn't > actually write the schema to the metastorage. > I believe that two things must be done in the scope of this ticket: > 1. Actually write new schema to the metastorage. > 2. Use metastorage's cluster time as the new schema's activation time. We > must also make use of Delay Duration which is described in the > https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
[
https://issues.apache.org/jira/browse/IGNITE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-19271:
Description:
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
h3. Problem
We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
not straightforward. This is because we have too many actors:
* There's a {_}client{_}, that chooses Tu, because it's the only actor that
can affect message content.
* There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses Tm.
* There's everybody else, who expect a correspondence between Tu and Tm.
First two actors are important, because they have independent clocks, but must
coordinate the same event. This is impossible with described protocol.
h3. Discussion
Let's consider these two solutions:
# Client generates Tm.
# Meta-storage generates Tu.
Option 1 is out of question, there must be only a single node at any given
moment in time, that's responsible for the linear order of time in messages.
What about option 2? Since meta-storage doesn't know anything about commands
semantics, it can't really generate any data. So this solution doesn't work
either.
h3. Solution
Combined solution could be the following:
* Client sends DD as part of the command (this is not a constant, user _can_
configure it, if they really feel like doing it)
* Meta-storage generates {{Tm}}
* Every node, upon receiving the update, calculates {{Tu}}
This could work, if nodes would have never been restarted. There's one problem
that needs to be solved: recovering the values of {{Tm}} from the (old) data
upon node restart.
This can be achieved by persisting safeTime along with revision as a part of
metadata, that can be retrieved back through the meta-storage service API.
In other words:
1. Client sends
{code:java}
schema.latest = 5
schema.5.data = ...
schema.5.dd = 30s{code}
2. Lease-holder adds meta-data to the command:
{code:java}
safeTime = 10:10
{code}
3. Meta-storage listener writes the data:
{code:java}
revision = 33
schema.latest = 5
schema.5.data = ...
schema.5.dd = 30s
revision.33.safeTime = 10:10:00{code}
How can you read {{{}Tu{}}}:
* read "{{{}schema.5.dd"{}}};
* read its revision, it's 33;
* read a timestamp of revision 33 via specialized API;
* add two values together.
h3. Implications and restrictions
There's a cleanup process in the meta-storage. It will eventually remove any
"revision.x.safeTime" values, because corresponding revision became obsolete.
But, we should somehow preserve timestamps of revisions that are used by
schemas. Such behaviour can be achieved, if components can reserve a revision,
and meta-storage can't compact it unless the reservation has been revoked.
was:
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
h3. Problem
We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
not straightforward. This is because we have too many actors:
* There's a {_}client{_}, that chooses Tu, because it's the only actor that
can affect message content.
* There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses Tm.
* There's everybody else, who expect a correspondence between Tu and Tm.
First two actors are important, because they have independent clocks, but must
coordinate the same event. This is impossible with described protocol.
h3. Discussion
Let's consider these two solutions:
# Client generates Tm.
# Meta-storage generates Tu.
Option 1 is out of question, there must be only a single node at any given
moment in time, that's responsible for the linear order of time in messages.
What about option 2? Since meta-storage doesn't know anything about commands
semantics, it can't really generate any data. So this solution doesn't work
either.
h3. Solution
Combined solution could be the following:
* Client sends DD as part of the command (this is not a constant, user _can_
configure it, if they really feel like doing it)
* Meta-storage generates {{Tm}}
* Every node, upon receiving the update, calculates {{Tu}}
This could work, if nodes would have never been restarted. There's one problem
that needs to be solved: recovering the values of {{Tm}} from the (old) data
upon node restart.
This can be achieved by persisting safeTime along with revision as a part of
metadata, that can be retrieved back through the meta-storage service API.
In other words:
1. C
[jira] [Commented] (IGNITE-19185) Fix index destruction after index creation started
[
https://issues.apache.org/jira/browse/IGNITE-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710860#comment-17710860
]
Semyon Danilov commented on IGNITE-19185:
-
The patch looks awesome to me!
> Fix index destruction after index creation started
> --
>
> Key: IGNITE-19185
> URL: https://issues.apache.org/jira/browse/IGNITE-19185
> Project: Ignite
> Issue Type: Bug
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> It was found that if the destruction of the index occurs immediately after
> the start of its creation, then errors occur during its building:
> *org.apache.ignite.internal.sql.api.ItSqlSynchronousApiTest#ddl*
> {code:java}
> 2023-04-02 13:53:16:882 +0300 [INFO][%issat_n_0%build-index-0][IndexBuilder]
> Start building the index: [table=TEST,
> tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa, partitionId=14, index=TEST_IDX,
> indexId=55e899e3-5404-4409-8570-aad4459c2908]
> 2023-04-02 13:53:16:883 +0300 [INFO][%issat_n_0%build-index-6][IndexBuilder]
> Start building the index: [table=TEST,
> tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa, partitionId=8, index=TEST_IDX,
> indexId=55e899e3-5404-4409-8570-aad4459c2908]
> 2023-04-02 13:53:16:884 +0300 [INFO][vault0][IndexManager] Index dropped
> [schema=PUBLIC, index=TEST_IDX]
> 2023-04-02 13:53:16:885 +0300 [ERROR][%issat_n_0%build-index-6][IndexBuilder]
> Index build error: [table=TEST, tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa,
> partitionId=15, index=TEST_IDX, indexId=55e899e3-5404-4409-8570-aad4459c2908]
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:f9741480-3eaf-4c32-8220-66e9a9118d48 Index configuration for
> "55e899e3-5404-4409-8570-aad4459c2908" could not be found
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:f9741480-3eaf-4c32-8220-66e9a9118d48 Index configuration for
> "55e899e3-5404-4409-8570-aad4459c2908" could not be found
> at
> org.apache.ignite.internal.storage.engine.MvTableStorage.getOrCreateIndex(MvTableStorage.java:94)
> at
> org.apache.ignite.internal.index.IndexBuilder$BuildIndexTask.collectRowIdBatch(IndexBuilder.java:258)
> at
> org.apache.ignite.internal.index.IndexBuilder$BuildIndexTask.lambda$run$1(IndexBuilder.java:164)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> ... 4 more
> 2023-04-02 13:53:16:883 +0300 [INFO][%issat_n_0%build-index-2][IndexBuilder]
> Start building the index: [table=TEST,
> tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa, partitionId=1, index=TEST_IDX,
> indexId=55e899e3-5404-4409-8570-aad4459c2908]
> 2023-04-02 13:53:16:887 +0300 [ERROR][%issat_n_0%build-index-6][IndexBuilder]
> Index build error: [table=TEST, tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa,
> partitionId=18, index=TEST_IDX, indexId=55e899e3-5404-4409-8570-aad4459c2908]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19220) Prohibit marking NetworkMessage with Marshallable
[ https://issues.apache.org/jira/browse/IGNITE-19220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19220: Description: ActionRequest's command is marked as Marshallable and at the same time, command is a NetworkMessage. This is obsolete. Annotation processor should fail compilation in such cases. > Prohibit marking NetworkMessage with Marshallable > - > > Key: IGNITE-19220 > URL: https://issues.apache.org/jira/browse/IGNITE-19220 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > ActionRequest's command is marked as Marshallable and at the same time, > command is a NetworkMessage. This is obsolete. Annotation processor should > fail compilation in such cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19220) Prohibit marking NetworkMessage with Marshallable
Semyon Danilov created IGNITE-19220: --- Summary: Prohibit marking NetworkMessage with Marshallable Key: IGNITE-19220 URL: https://issues.apache.org/jira/browse/IGNITE-19220 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov Assignee: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19199) Idle safe time propagation for the metastorage
[ https://issues.apache.org/jira/browse/IGNITE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19199: Labels: ignite-3 (was: ) > Idle safe time propagation for the metastorage > -- > > Key: IGNITE-19199 > URL: https://issues.apache.org/jira/browse/IGNITE-19199 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > > In https://issues.apache.org/jira/browse/IGNITE-19028 safe time is propagated > from the leader every 1 second, which is sub-optimal. The timeout should be > configurable + safe time should be only propagated if the metastorage is > really idle (no updates were made in this timeout period) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19199) Idle safe time propagation for the metastorage
[ https://issues.apache.org/jira/browse/IGNITE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19199: Description: In https://issues.apache.org/jira/browse/IGNITE-19028 safe time is propagated from the leader every 1 second, which is sub-optimal. The timeout should be configurable + safe time should be only propagated if the metastorage is really idle (no updates were made in this timeout period) > Idle safe time propagation for the metastorage > -- > > Key: IGNITE-19199 > URL: https://issues.apache.org/jira/browse/IGNITE-19199 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > > In https://issues.apache.org/jira/browse/IGNITE-19028 safe time is propagated > from the leader every 1 second, which is sub-optimal. The timeout should be > configurable + safe time should be only propagated if the metastorage is > really idle (no updates were made in this timeout period) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19199) Idle safe time propagation for the metastorage
Semyon Danilov created IGNITE-19199: --- Summary: Idle safe time propagation for the metastorage Key: IGNITE-19199 URL: https://issues.apache.org/jira/browse/IGNITE-19199 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19028) Implement safe-time propagation for meta-storage raft-group
[ https://issues.apache.org/jira/browse/IGNITE-19028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-19028: --- Assignee: Semyon Danilov > Implement safe-time propagation for meta-storage raft-group > --- > > Key: IGNITE-19028 > URL: https://issues.apache.org/jira/browse/IGNITE-19028 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > For the future implementation of schema-synchronization, we need to have a > hybrid-timestamp, associated with the meta-storage. > Database schema changes are always associated with time, and proper place to > store them would be a meta-storage. > We don't have an "partition replica listener" that would have been a single > source of truth when it comes to new "write" commands. In case of > meta-storage, all nodes may create write commands. Assigning a time from the > _hlc_ wouldn't work - there's a chance of having out-of order events, which > is really, really bad. > In other words, timestamps should come in order. Does this mean that > meta-storage should also have its own replica listener? That's one > possibility. > Another possibility is to make leader into a timestamp-generator. This would > lead to changes in JRaft code, but still, this may be the right way to go. It > simply requires less changes to the code. We should just remember to adjust > the clock on leader's re-election, so that time would be monotonic. > By the way, if we go with the second option, it would also fit safe time > propagation in partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19070) Remove stale TODO of IGNITE-16081
[ https://issues.apache.org/jira/browse/IGNITE-19070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703084#comment-17703084 ] Semyon Danilov commented on IGNITE-19070: - Thank you for the review, merged to the main branch > Remove stale TODO of IGNITE-16081 > - > > Key: IGNITE-19070 > URL: https://issues.apache.org/jira/browse/IGNITE-19070 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 20m > Remaining Estimate: 0h > > Remove TODO IGNITE-16081 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19070) Remove stale TODO of IGNITE-16081
[ https://issues.apache.org/jira/browse/IGNITE-19070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19070: Description: Remove TODO IGNITE-16081 (was: Remove TODO ) > Remove stale TODO of IGNITE-16081 > - > > Key: IGNITE-19070 > URL: https://issues.apache.org/jira/browse/IGNITE-19070 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > Remove TODO IGNITE-16081 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19070) Remove stale TODO of IGNITE-16081
[ https://issues.apache.org/jira/browse/IGNITE-19070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-19070: Description: Remove TODO > Remove stale TODO of IGNITE-16081 > - > > Key: IGNITE-19070 > URL: https://issues.apache.org/jira/browse/IGNITE-19070 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > Remove TODO -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19070) Remove stale TODO of IGNITE-16081
Semyon Danilov created IGNITE-19070: --- Summary: Remove stale TODO of IGNITE-16081 Key: IGNITE-19070 URL: https://issues.apache.org/jira/browse/IGNITE-19070 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov Assignee: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16627) SNI extension is missing when Java thin client is connecting to Ignite cluster with SSL enabled
[
https://issues.apache.org/jira/browse/IGNITE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701724#comment-17701724
]
Semyon Danilov commented on IGNITE-16627:
-
The problem:
There is a SSL gateway proxy in k8s cluster. In order to connect to a node of
the cluster, SNI extension must be provided. Otherwise SSL gateway doesn’t know
where to route incoming traffic.
A thin client is located outside of the k8s cluster. It is aware of some
address (probably the address of the gateway). As a WA a custom SSLFactory
might be implemented, which provides SNI extension. However, this way thin
client can only connect to one node (the one whose Server Name is provided). So
when a client learns (through affinity awareness) of other nodes, it won’t be
able to connect to them, outgoing traffic will always be routed to the first
node with provided Server Name. This means that there’s a single point of
denial and that load will be unbalanced.
Implementation problem(s):
According to the TLS protocol, we can only set one Server Name per connection.
ATM we have one SSLContext per all connections. Looks like we need to have a
context per connection. By default, we should set each address (if it is a
hostname) from ClientConfiguration as a Server Name for a connection to that
address. In case we are learning about a node “in runtime” – we should cycle
through it’s addresses and use hostname as Server Name for connections. There
can be a problem if we get an IP of a node (that will probably be the internal
k8s cluster IP) – we most likely won’t be able to connect to it. However we
should be able to connect by hostname
> SNI extension is missing when Java thin client is connecting to Ignite
> cluster with SSL enabled
> ---
>
> Key: IGNITE-16627
> URL: https://issues.apache.org/jira/browse/IGNITE-16627
> Project: Ignite
> Issue Type: Bug
> Components: thin client
>Affects Versions: 2.11
>Reporter: Maria Makedonskaya
>Priority: Major
>
> Motivation: There are cases then ignite clients are connecting to a cluster
> which is located inside Kubernetes(k8s) and k8s cluster has an ingress
> gateway that routes TLS traffic using SNI extension.
> Need to provide hostnames from
> org.apache.ignite.configuration.ClientConfiguration#setAddresses to SNI
> extention.
> SSLContext for java thin client is creating in
> org.apache.ignite.internal.client.thin.ClientSslUtils#getSslContext. Possibly
> we can use org.apache.ignite.ssl.SSLContextWrapper there to provide
> additional SSLParameters(like it's done in
> org.apache.ignite.ssl.SslContextFactory#createSslContext). For adding SNI
> extension need to add hostnames via
> javax.net.ssl.SSLParameters#setServerNames.
> Also need to check that other thin clients and thick clients add SNI to
> handshake.
> Possibly in
> org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter#onSessionOpened we
> need additionally to replace
> from:
> {code:java}
> engine = this.sslCtx.createSSLEngine();{code}
> to:
> {code:java}
> engine = this.sslCtx.createSSLEngine(ses.remoteAddress().getHostName(),
> ses.remoteAddress().getPort()){code}
> In this case, if an IP address is set to ClientConfiguration#setAddresses
> then SNI extension will be added with reverse lookup hostname. If hostname
> with a port is set to ClientConfiguration#setAddresses no SNI extension will
> be added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-19023) Fix TODO mentioning IGNITE-16526
[
https://issues.apache.org/jira/browse/IGNITE-19023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700524#comment-17700524
]
Semyon Danilov commented on IGNITE-19023:
-
LGTM. Thank you for the contribution, merged to the main branch
> Fix TODO mentioning IGNITE-16526
>
>
> Key: IGNITE-19023
> URL: https://issues.apache.org/jira/browse/IGNITE-19023
> Project: Ignite
> Issue Type: Task
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ItClusterCommandTest#initClusterWithNodesOfDifferentRoles has the following
> TODO:
> {code:java}
> // TODO: when IGNITE-16526 is implemented, also check that the logical
> topology contains all 4 nodes
> {code}
> Since IGNITE-1652 is already closed, this TODO must be resolved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (IGNITE-18743) Prepare the first version of a design for schema synchronization
[ https://issues.apache.org/jira/browse/IGNITE-18743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699980#comment-17699980 ] Semyon Danilov edited comment on IGNITE-18743 at 3/14/23 6:38 AM: -- The result can be found here https://cwiki.apache.org/confluence/display/IGNITE/IEP-98%3A+Schema+Synchronization was (Author: sdanilov): The result can be found here > Prepare the first version of a design for schema synchronization > > > Key: IGNITE-18743 > URL: https://issues.apache.org/jira/browse/IGNITE-18743 > Project: Ignite > Issue Type: Task > Components: persistence >Reporter: Sergey Chugunov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > The first version has to give an overall view of how schema synchronization > will work and how the most important cases will be addressed (like multiple > versions of schemas, concurrent user ops and so on). > Some ideas are explained (very briefly though) in > [IEP-91|https://cwiki.apache.org/confluence/display/IGNITE/IEP-91:+Transaction+protocol#IEP91:Transactionprotocol-Lock-freeRWtransactions] > document (see *Application to metadata management* section), we need to > clarify them and cover with more details. > As a result of the task should be a document and a set of several > ready-for-development tasks to implement at least happy-cases of the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18743) Prepare the first version of a design for schema synchronization
[ https://issues.apache.org/jira/browse/IGNITE-18743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699980#comment-17699980 ] Semyon Danilov commented on IGNITE-18743: - The result can be found here > Prepare the first version of a design for schema synchronization > > > Key: IGNITE-18743 > URL: https://issues.apache.org/jira/browse/IGNITE-18743 > Project: Ignite > Issue Type: Task > Components: persistence >Reporter: Sergey Chugunov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > The first version has to give an overall view of how schema synchronization > will work and how the most important cases will be addressed (like multiple > versions of schemas, concurrent user ops and so on). > Some ideas are explained (very briefly though) in > [IEP-91|https://cwiki.apache.org/confluence/display/IGNITE/IEP-91:+Transaction+protocol#IEP91:Transactionprotocol-Lock-freeRWtransactions] > document (see *Application to metadata management* section), we need to > clarify them and cover with more details. > As a result of the task should be a document and a set of several > ready-for-development tasks to implement at least happy-cases of the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18743) Prepare the first version of a design for schema synchronization
[ https://issues.apache.org/jira/browse/IGNITE-18743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-18743: --- Assignee: Semyon Danilov (was: Roman Puchkovskiy) > Prepare the first version of a design for schema synchronization > > > Key: IGNITE-18743 > URL: https://issues.apache.org/jira/browse/IGNITE-18743 > Project: Ignite > Issue Type: Task > Components: persistence >Reporter: Sergey Chugunov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > The first version has to give an overall view of how schema synchronization > will work and how the most important cases will be addressed (like multiple > versions of schemas, concurrent user ops and so on). > Some ideas are explained (very briefly though) in > [IEP-91|https://cwiki.apache.org/confluence/display/IGNITE/IEP-91:+Transaction+protocol#IEP91:Transactionprotocol-Lock-freeRWtransactions] > document (see *Application to metadata management* section), we need to > clarify them and cover with more details. > As a result of the task should be a document and a set of several > ready-for-development tasks to implement at least happy-cases of the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18968) Possible race between updating a low watermark and processing the last batch for storage in a background GC
[ https://issues.apache.org/jira/browse/IGNITE-18968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17697293#comment-17697293 ] Semyon Danilov commented on IGNITE-18968: - LGTM > Possible race between updating a low watermark and processing the last batch > for storage in a background GC > --- > > Key: IGNITE-18968 > URL: https://issues.apache.org/jira/browse/IGNITE-18968 > Project: Ignite > Issue Type: Bug >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > After some analysis of the background garbage collector code, a possible race > was found between updating the low watermark and processing the last batch of > storage. > Needs to be fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18970) Improve work directory extension's keep capability
[ https://issues.apache.org/jira/browse/IGNITE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-18970: --- Assignee: Semyon Danilov > Improve work directory extension's keep capability > -- > > Key: IGNITE-18970 > URL: https://issues.apache.org/jira/browse/IGNITE-18970 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > WorkDirectory extension should allow user to preserve work directories of > specific tests and, if needed, move work directories to specific folders. > That would be useful for teamcity, because we can use a single folder for > artifacts to store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18970) Improve work directory extension's keep capability
Semyon Danilov created IGNITE-18970: --- Summary: Improve work directory extension's keep capability Key: IGNITE-18970 URL: https://issues.apache.org/jira/browse/IGNITE-18970 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov WorkDirectory extension should allow user to preserve work directories of specific tests and, if needed, move work directories to specific folders. That would be useful for teamcity, because we can use a single folder for artifacts to store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18882) Fix tombstone is stored if it is the first entry of version chain
[
https://issues.apache.org/jira/browse/IGNITE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov reassigned IGNITE-18882:
---
Assignee: Semyon Danilov
> Fix tombstone is stored if it is the first entry of version chain
> -
>
> Key: IGNITE-18882
> URL: https://issues.apache.org/jira/browse/IGNITE-18882
> Project: Ignite
> Issue Type: Bug
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> This test in AbstractMvPartitionStorageGcTest should pass
> {code:java}
> void testTombstoneFirst() {
> addAndCommit(null);
> addAndCommit(TABLE_ROW);
> addAndCommit(TABLE_ROW2);
> BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
> assertRowMatches(row.binaryRow(), TABLE_ROW);
> }
> {code}
> At this moment, storages will store the tombstone if it is the first
> committed value which disrupts the GC flow.
> addWrite with null argument as a first version of row is valid, for example:
> {code:sql}
> CREATE TABLE test (
> id INT
> );
> BEGIN;
> INSERT INTO test VALUES(1);
> DELETE from test where id = 1;
> COMMIT;
> {code}
> is ok, but tombstone should not be stored (so the operation should be no-op)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18909) Pass GC low watermark for cooperative GC
[
https://issues.apache.org/jira/browse/IGNITE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-18909:
Description:
IGNITE-18033 introduced 2 new method overloads in the {{StorageUpdateHandler}}
with a new argument -- GC low watermark. This is needed for the cooperative GC
(GC that is done just before the storage update). We need to make use of this
arguments in the actual business logic
> Pass GC low watermark for cooperative GC
>
>
> Key: IGNITE-18909
> URL: https://issues.apache.org/jira/browse/IGNITE-18909
> Project: Ignite
> Issue Type: Improvement
>Reporter: Semyon Danilov
>Priority: Major
> Labels: ignite-3
>
> IGNITE-18033 introduced 2 new method overloads in the
> {{StorageUpdateHandler}} with a new argument -- GC low watermark. This is
> needed for the cooperative GC (GC that is done just before the storage
> update). We need to make use of this arguments in the actual business logic
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18909) Pass GC low watermark for cooperative GC
[ https://issues.apache.org/jira/browse/IGNITE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-18909: Labels: ignite-3 (was: ) > Pass GC low watermark for cooperative GC > > > Key: IGNITE-18909 > URL: https://issues.apache.org/jira/browse/IGNITE-18909 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18909) Pass GC low watermark for cooperative GC
[ https://issues.apache.org/jira/browse/IGNITE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-18909: Epic Link: IGNITE-17571 > Pass GC low watermark for cooperative GC > > > Key: IGNITE-18909 > URL: https://issues.apache.org/jira/browse/IGNITE-18909 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18909) Pass GC low watermark for cooperative GC
Semyon Danilov created IGNITE-18909: --- Summary: Pass GC low watermark for cooperative GC Key: IGNITE-18909 URL: https://issues.apache.org/jira/browse/IGNITE-18909 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (IGNITE-18903) Remove TODOs mentioning closed issues
[ https://issues.apache.org/jira/browse/IGNITE-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693328#comment-17693328 ] Semyon Danilov edited comment on IGNITE-18903 at 2/25/23 4:34 AM: -- LGTM. Thank you for the contribution, merged to the main branch. was (Author: sdanilov): LGTM > Remove TODOs mentioning closed issues > - > > Key: IGNITE-18903 > URL: https://issues.apache.org/jira/browse/IGNITE-18903 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 20m > Remaining Estimate: 0h > > The following issues are already closed, but still mentioned in the code: > IGNITE-18505 > IGNITE-17775 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18033) Implement cooperative GC of MV data during RAFT commands execution
[ https://issues.apache.org/jira/browse/IGNITE-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693230#comment-17693230 ] Semyon Danilov commented on IGNITE-18033: - As we decided to implement a GC via a queue, this particular task should be implemented as a simple GC call (preferably in batches) whenever writes are made to the partition storage > Implement cooperative GC of MV data during RAFT commands execution > -- > > Key: IGNITE-18033 > URL: https://issues.apache.org/jira/browse/IGNITE-18033 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > Please refer to the Epic (IGNITE-17571) for the basic description. Please > also refer to IGNITE-18031 for naive implementation details and general > thoughts. > Technically, there is a possibility that the background GC process wouldn't > catch up if there's too much data being loaded into the system. Scanning > through the entire partition takes time, and only a small subset of data > could be under a constant stream of modification. > To account for that, each update can be preceded with the manual GC of that > row. In this case, there's less work for the background processor, and > there's an empirical sense that frequently updated data will be just as > frequently vacuumed, thus not allowing too much garbage to appear in the > first place. > Now that's not a guarantee that there would be no problems at all, so we > should also think about other ways of cooperative GC. > Of course, it would be nice to have a queue of "old" rows ready for you to > clean. How feasible is it? It would have to be supported by every engine. > That's extra data and extra time for management. Looks pretty much exactly > like a TTL manager in Ignite 2.x. I assume that we don't implement anything > like that right now, starting with more naive approaches. Anyway, please > refer to the IGNITE-18113 to see more details, maybe that's the way to go. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18033) Implement cooperative GC of MV data during RAFT commands execution
[ https://issues.apache.org/jira/browse/IGNITE-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-18033: --- Assignee: Semyon Danilov > Implement cooperative GC of MV data during RAFT commands execution > -- > > Key: IGNITE-18033 > URL: https://issues.apache.org/jira/browse/IGNITE-18033 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > Please refer to the Epic (IGNITE-17571) for the basic description. Please > also refer to IGNITE-18031 for naive implementation details and general > thoughts. > Technically, there is a possibility that the background GC process wouldn't > catch up if there's too much data being loaded into the system. Scanning > through the entire partition takes time, and only a small subset of data > could be under a constant stream of modification. > To account for that, each update can be preceded with the manual GC of that > row. In this case, there's less work for the background processor, and > there's an empirical sense that frequently updated data will be just as > frequently vacuumed, thus not allowing too much garbage to appear in the > first place. > Now that's not a guarantee that there would be no problems at all, so we > should also think about other ways of cooperative GC. > Of course, it would be nice to have a queue of "old" rows ready for you to > clean. How feasible is it? It would have to be supported by every engine. > That's extra data and extra time for management. Looks pretty much exactly > like a TTL manager in Ignite 2.x. I assume that we don't implement anything > like that right now, starting with more naive approaches. Anyway, please > refer to the IGNITE-18113 to see more details, maybe that's the way to go. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18565) Modify getOrCreateMvPartition and getMvPartition of MvTableStorage to return the future
[ https://issues.apache.org/jira/browse/IGNITE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692782#comment-17692782 ] Semyon Danilov commented on IGNITE-18565: - LGTM! > Modify getOrCreateMvPartition and getMvPartition of MvTableStorage to return > the future > --- > > Key: IGNITE-18565 > URL: https://issues.apache.org/jira/browse/IGNITE-18565 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > We need to make *MvTableStorage#getOrCreateMvPartition* and > *MvTableStorage#getMvPartition* return the > *CompletableFuture*. > Since, for example, we need to wait for the partition to be deleted before > creating / recreating it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18033) Implement cooperative GC of MV data during RAFT commands execution
[ https://issues.apache.org/jira/browse/IGNITE-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-18033: Description: Please refer to the Epic (IGNITE-17571) for the basic description. Please also refer to IGNITE-18031 for naive implementation details and general thoughts. Technically, there is a possibility that the background GC process wouldn't catch up if there's too much data being loaded into the system. Scanning through the entire partition takes time, and only a small subset of data could be under a constant stream of modification. To account for that, each update can be preceded with the manual GC of that row. In this case, there's less work for the background processor, and there's an empirical sense that frequently updated data will be just as frequently vacuumed, thus not allowing too much garbage to appear in the first place. Now that's not a guarantee that there would be no problems at all, so we should also think about other ways of cooperative GC. Of course, it would be nice to have a queue of "old" rows ready for you to clean. How feasible is it? It would have to be supported by every engine. That's extra data and extra time for management. Looks pretty much exactly like a TTL manager in Ignite 2.x. I assume that we don't implement anything like that right now, starting with more naive approaches. Anyway, please refer to the IGNITE-18113 to see more details, maybe that's the way to go. was: Please refer to the Epic for the basic description. Please also refer to IGNITE-18031 for naive implementation details and general thoughts. Technically, there is a possibility that the background GC process wouldn't catch up if there's too much data being loaded into the system. Scanning through the entire partition takes time, and only a small subset of data could be under a constant stream of modification. To account for that, each update can be preceded with the manual GC of that row. In this case, there's less work for the background processor, and there's an empirical sense that frequently updated data will be just as frequently vacuumed, thus not allowing too much garbage to appear in the first place. Now that's not a guarantee that there would be no problems at all, so we should also think about other ways of cooperative GC. Of course, it would be nice to have a queue of "old" rows ready for you to clean. How feasible is it? It would have to be supported by every engine. That's extra data and extra time for management. Looks pretty much exactly like a TTL manager in Ignite 2.x. I assume that we don't implement anything like that right now, starting with more naive approaches. Anyway, please refer to the IGNITE-18113 to see more details, maybe that's the way to go. > Implement cooperative GC of MV data during RAFT commands execution > -- > > Key: IGNITE-18033 > URL: https://issues.apache.org/jira/browse/IGNITE-18033 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Priority: Major > Labels: ignite-3 > > Please refer to the Epic (IGNITE-17571) for the basic description. Please > also refer to IGNITE-18031 for naive implementation details and general > thoughts. > Technically, there is a possibility that the background GC process wouldn't > catch up if there's too much data being loaded into the system. Scanning > through the entire partition takes time, and only a small subset of data > could be under a constant stream of modification. > To account for that, each update can be preceded with the manual GC of that > row. In this case, there's less work for the background processor, and > there's an empirical sense that frequently updated data will be just as > frequently vacuumed, thus not allowing too much garbage to appear in the > first place. > Now that's not a guarantee that there would be no problems at all, so we > should also think about other ways of cooperative GC. > Of course, it would be nice to have a queue of "old" rows ready for you to > clean. How feasible is it? It would have to be supported by every engine. > That's extra data and extra time for management. Looks pretty much exactly > like a TTL manager in Ignite 2.x. I assume that we don't implement anything > like that right now, starting with more naive approaches. Anyway, please > refer to the IGNITE-18113 to see more details, maybe that's the way to go. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18882) Fix tombstone is stored if it is the first entry of version chain
[
https://issues.apache.org/jira/browse/IGNITE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-18882:
Description:
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow.
addWrite with null argument as a first version of row is valid, for example:
{code:sql}
CREATE TABLE test (
id INT
);
BEGIN;
INSERT INTO test VALUES(1);
DELETE from test where id = 1;
COMMIT;
{code}
is ok, but tombstone should not be stored (so the operation should be no-op)
was:
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow.
addWrite with null argument as a first version of row is valid, for example:
{code:sql}
CREATE TABLE test (id INT);
DELETE from test where id = 0;
{code}
is ok, but tombstone should not be stored (so the operation should be no-op)
> Fix tombstone is stored if it is the first entry of version chain
> -
>
> Key: IGNITE-18882
> URL: https://issues.apache.org/jira/browse/IGNITE-18882
> Project: Ignite
> Issue Type: Bug
>Reporter: Semyon Danilov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> This test in AbstractMvPartitionStorageGcTest should pass
> {code:java}
> void testTombstoneFirst() {
> addAndCommit(null);
> addAndCommit(TABLE_ROW);
> addAndCommit(TABLE_ROW2);
> BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
> assertRowMatches(row.binaryRow(), TABLE_ROW);
> }
> {code}
> At this moment, storages will store the tombstone if it is the first
> committed value which disrupts the GC flow.
> addWrite with null argument as a first version of row is valid, for example:
> {code:sql}
> CREATE TABLE test (
> id INT
> );
> BEGIN;
> INSERT INTO test VALUES(1);
> DELETE from test where id = 1;
> COMMIT;
> {code}
> is ok, but tombstone should not be stored (so the operation should be no-op)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18882) Fix tombstone is stored if it is the first entry of version chain
[
https://issues.apache.org/jira/browse/IGNITE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-18882:
Description:
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow.
addWrite with null argument as a first version of row is valid, for example:
{code:sql}
CREATE TABLE test (id INT);
DELETE from test where id = 0;
{code}
is ok, but tombstone should not be stored (so the operation should be no-op)
was:
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow.
addWrite with null argument as a first version of row is valid, for example:
{code:sql}
CREATE TABLE test (id INT);
DELETE from test where id = 0;
{code}
is ok. But tombstone should not be stored.
> Fix tombstone is stored if it is the first entry of version chain
> -
>
> Key: IGNITE-18882
> URL: https://issues.apache.org/jira/browse/IGNITE-18882
> Project: Ignite
> Issue Type: Bug
>Reporter: Semyon Danilov
>Priority: Major
> Labels: ignite-3
>
> This test in AbstractMvPartitionStorageGcTest should pass
> {code:java}
> void testTombstoneFirst() {
> addAndCommit(null);
> addAndCommit(TABLE_ROW);
> addAndCommit(TABLE_ROW2);
> BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
> assertRowMatches(row.binaryRow(), TABLE_ROW);
> }
> {code}
> At this moment, storages will store the tombstone if it is the first
> committed value which disrupts the GC flow.
> addWrite with null argument as a first version of row is valid, for example:
> {code:sql}
> CREATE TABLE test (id INT);
> DELETE from test where id = 0;
> {code}
> is ok, but tombstone should not be stored (so the operation should be no-op)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18882) Fix tombstone is stored if it is the first entry of version chain
[
https://issues.apache.org/jira/browse/IGNITE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-18882:
Description:
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow.
addWrite with null argument as a first version of row is valid, for example:
{code:sql}
CREATE TABLE test (id INT);
DELETE from test where id = 0;
{code}
is ok. But tombstone should not be stored.
was:
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow
> Fix tombstone is stored if it is the first entry of version chain
> -
>
> Key: IGNITE-18882
> URL: https://issues.apache.org/jira/browse/IGNITE-18882
> Project: Ignite
> Issue Type: Bug
>Reporter: Semyon Danilov
>Priority: Major
> Labels: ignite-3
>
> This test in AbstractMvPartitionStorageGcTest should pass
> {code:java}
> void testTombstoneFirst() {
> addAndCommit(null);
> addAndCommit(TABLE_ROW);
> addAndCommit(TABLE_ROW2);
> BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
> assertRowMatches(row.binaryRow(), TABLE_ROW);
> }
> {code}
> At this moment, storages will store the tombstone if it is the first
> committed value which disrupts the GC flow.
> addWrite with null argument as a first version of row is valid, for example:
> {code:sql}
> CREATE TABLE test (id INT);
> DELETE from test where id = 0;
> {code}
> is ok. But tombstone should not be stored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-18882) Fix tombstone is stored if it is the first entry of version chain
Semyon Danilov created IGNITE-18882:
---
Summary: Fix tombstone is stored if it is the first entry of
version chain
Key: IGNITE-18882
URL: https://issues.apache.org/jira/browse/IGNITE-18882
Project: Ignite
Issue Type: Bug
Reporter: Semyon Danilov
This test in AbstractMvPartitionStorageGcTest should pass
{code:java}
void testTombstoneFirst() {
addAndCommit(null);
addAndCommit(TABLE_ROW);
addAndCommit(TABLE_ROW2);
BinaryRowAndRowId row = pollForVacuum(HybridTimestamp.MAX_VALUE);
assertRowMatches(row.binaryRow(), TABLE_ROW);
}
{code}
At this moment, storages will store the tombstone if it is the first committed
value which disrupts the GC flow
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-18739) Cleanup indexes as part of garbage collection
[ https://issues.apache.org/jira/browse/IGNITE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov reassigned IGNITE-18739: --- Assignee: Semyon Danilov > Cleanup indexes as part of garbage collection > - > > Key: IGNITE-18739 > URL: https://issues.apache.org/jira/browse/IGNITE-18739 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > > To maintain the integrity of the partition storage, it is necessary to > implement a mechanism for cleaning up indexes during the garbage collection > process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
