[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230208#comment-15230208 ] Semen Boikov commented on IGNITE-2864: -- Hi Anton, I reviewed your changes, have some comments: - in atomic cache when store is updated on backup you need check 'skipStore' flag (it should be passed in GridDhtAtomicUpdateRequest) - method IgniteTxRemoteStateAdapter.stores() should return only stores participating in transaction. When fix it, please try to avoid unnesessary iterations if we know in advance that caches participating in tx do not have stores configured. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239314#comment-15239314 ] Anton Vinogradov commented on IGNITE-2864: -- Semen, I fixed issues you've described, please check. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239909#comment-15239909 ] Semen Boikov commented on IGNITE-2864: -- Hi Anton, I have some comments: - I think correct way to resolve conflict is call CacheVersionConflictResolver since it can do more then just version comparison - you added 'flags' in GridDhtAtomicUpdateRequest but did not change writeTo/readFrom methods, please fix and add test - why in IgniteTxRemoteSingleStateImpl.stores you create HashSet with capacity 3 if you know in advance that there is only one store? - please add tests with near cache and make sure 'batchStoreCommit' is not called for near cache txs - I think it is possible to implement IgniteTxRemoteStateImpl.stores more efficiently (do not iterate over read entries, avoid any collections creations if there are no stores, etc). Please also run some tx putAll benchmarks in FULL_SYNC mode with backup to make sure there is no performance drop. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241371#comment-15241371 ] Anton Vinogradov commented on IGNITE-2864: -- Semen, Please check changes. I'll check benchmarks in case everything is ok. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244076#comment-15244076 ] Anton Vinogradov commented on IGNITE-2864: -- Partially fixed found on review. Main problem is conflict resolving in case of local store usage. This problem exists at current implementation too. I think that this problem should be solved different from decided on review. Env: Each node has local store. Local store contains primary and backup partitions. At node fail store can be used to restore entries. Problem: Cluster partially failed, amount of failed node > backups. Initial solution: Restart failed nodes and load entries from local stores after restart. Resolve conflicts at rebalancing and user requests. Cons: A lot of changes required. Difficult to cover all cases. Better solution: Topology validator should be used to prevent work with inconsistent data. Recover steps: 1) Deny all user requests (admins should do that) 2) Restart all failed nodes. 3) Load all data from all available local stores at stable topology (after final rebalancing finished). All conflicts will be resovled using conflict resolver in this case, correct? All entries will be restored since we have backups at local stores. (in case lost stores <= backups) 4) Allow user requests. Thoughts? > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246952#comment-15246952 ] Dmitriy Setrakyan commented on IGNITE-2864: --- Is this feature optional? I think it should be set through configuration and should be turned off by default, because it only makes sense for per-JVM CacheStore and not for centralized databases. If it has been made configurable, what is the name of the configuration property? > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247866#comment-15247866 ] Anton Vinogradov commented on IGNITE-2864: -- Dmitriy, As far as I understand we'd like to keep compatibility with previous solution? I've added flag CacheConfiguration#isLocalStoreUpdateBackups and set it false by default. Maybe we have to use System property instead of flag at CacheConfiguration? > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249259#comment-15249259 ] Semen Boikov commented on IGNITE-2864: -- Reviewed, did few minor changes, good to merge (if everybody agree about this new flag 'isLocalStoreUpdateBackups '). > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250094#comment-15250094 ] Anton Vinogradov commented on IGNITE-2864: -- Eventually I've added system property IGNITE_LOCAL_STORE_KEEPS_PRIMARY_ONLY and set it false by default. It means that new behavior will be active by default. Also I added more tests for old and new behavior. In case user will try to combine nodes with versions "before" and "after" fix with configured local store he will gain warning: "Since Ignite 1.5.15 Local Store keeps primary and backup partitions. To keep primary partitions only please set system property IGNITE_LOCAL_STORE_KEEPS_PRIMARY_ONLY to 'true'." > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[
https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250146#comment-15250146
]
Dmitriy Setrakyan commented on IGNITE-2864:
---
Can you explain why you decided to use a system property instead of regular
configuration property?
Also, I am not sure I like the name, because this property is not about the
local store, but about the cache store in general. I think it should be named
{{isCentralizedStore}} and should be {{true}} by default. If store is
centralized, then there is no need to write backups to the store. On the other
hand, if store is local to a node, then it should have both, primary and backup
copies for the purpose of recovery.
Do you agree?
> Need update local store from primary and backups
>
>
> Key: IGNITE-2864
> URL: https://issues.apache.org/jira/browse/IGNITE-2864
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Reporter: Semen Boikov
>Assignee: Anton Vinogradov
> Fix For: 1.6
>
>
> Now cache local store is updated only from primary nodes, this means that
> data can be lost if primary node is not re-started after crash. Need fix it
> and update store from primaries and backups if store is local (for both tx
> and atomic caches).
> This test should work:
> - cache with 1 backup, two server nodes
> - execute cache put for key K
> - stop both nodes
> - restart only node which was backup for K
> - load data from local sore, update for K should be restored
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251534#comment-15251534 ] Anton Vinogradov commented on IGNITE-2864: -- Dmitriy, This property is about local store usage only. It gives you two ways of local store behavoir: to keep backup partitions (new) or no (old). Nothing changed for regular stores. I discussed solution based on configuration property with Yakov and we've decided that only compatibility flag (system property) should be provided to allow riskless upgrades. Also, as I wrote before, we've decided to turn new behavior on by default, but inform users by warning that behavior changed. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254217#comment-15254217 ] Dmitriy Setrakyan commented on IGNITE-2864: --- Anton, how do you know if the CacheStore is local or not? > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254542#comment-15254542 ] Semen Boikov commented on IGNITE-2864: -- Dmitry, store is local if it has CacheLocalStore annotation. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274427#comment-15274427 ] Dmitriy Setrakyan commented on IGNITE-2864: --- Let me summarize the new behavior, just so we are on the same page: # All stores with `@CacheLocalStore` annotation will persist primary and backup copies by default. # This behavior can be turned off if a user adds `IGNITE_LOCAL_STORE_KEEPS_PRIMARY_ONLY` system property. # No new configuration properties have been added. If the above is correct, then I think this change makes sense. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278417#comment-15278417 ] Anton Vinogradov commented on IGNITE-2864: -- Correct. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278464#comment-15278464 ] Dmitriy Setrakyan commented on IGNITE-2864: --- Sounds good, please remember to update the documentation. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282345#comment-15282345 ] Dmitriy Setrakyan commented on IGNITE-2864: --- Would be very nice to get it into 1.6 release. Looks like the only thing that is pending is a review. > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282518#comment-15282518 ] Semen Boikov commented on IGNITE-2864: -- Reviewed, looks good (just do not forget to set correct GridCacheStoreManagerAdapter.LOCAL_STORE_KEEPS_PRIMARY_AND_BACKUPS_SINCE before merge). > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2864) Need update local store from primary and backups
[ https://issues.apache.org/jira/browse/IGNITE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282519#comment-15282519 ] Anton Vinogradov commented on IGNITE-2864: -- Sure, Thanks! > Need update local store from primary and backups > > > Key: IGNITE-2864 > URL: https://issues.apache.org/jira/browse/IGNITE-2864 > Project: Ignite > Issue Type: Bug > Components: cache >Reporter: Semen Boikov >Assignee: Anton Vinogradov > Fix For: 1.6 > > > Now cache local store is updated only from primary nodes, this means that > data can be lost if primary node is not re-started after crash. Need fix it > and update store from primaries and backups if store is local (for both tx > and atomic caches). > This test should work: > - cache with 1 backup, two server nodes > - execute cache put for key K > - stop both nodes > - restart only node which was backup for K > - load data from local sore, update for K should be restored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
