Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Murtadha Hubail has submitted this change and it was merged.

Change subject: Introduce CheckpointManager API
..


Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1380
Sonar-Qube: Jenkins 
Tested-by: Jenkins 
BAD: Jenkins 
Integration-Tests: Jenkins 
Reviewed-by: Murtadha Hubail 
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
D 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointObject.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
21 files changed, 741 insertions(+), 354 deletions(-)

Approvals:
  Murtadha Hubail: Looks good to me, approved
  Jenkins: Verified; No violations found; No violations found; Verified



diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
index a3ae9a0..b1ca062 100644
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
@@ -278,7 +278,7 @@
 lccm.register((ILifeCycleComponent) datasetLifecycleManager);
 lccm.register((ILifeCycleComponent) 
txnSubsystem.getTransactionManager());
 lccm.register((ILifeCycleComponent) txnSubsystem.getLockManager());
-
+lccm.register(txnSubsystem.getCheckpointManager());
 }
 
 @Override
diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
index 1887f2e..8998c6b 100644
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
@@ -37,6 +37,7 @@
 import org.apache.asterix.common.config.IPropertiesProvider;
 import org.apache.asterix.common.config.MessagingProperties;
 import 

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Murtadha Hubail has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 9: Code-Review+2

Forwarding Ian's +2 after rebasing.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 9: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1336/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 9: BAD+1

BAD Compatibility Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/63/ : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 8:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3573/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 8
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Hello Ian Maxon, Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1380

to look at the new patch set (#8).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
D 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointObject.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
21 files changed, 741 insertions(+), 354 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/8
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 8
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 7:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3572/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 7
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Hello Ian Maxon, Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1380

to look at the new patch set (#7).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
D 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointObject.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
21 files changed, 740 insertions(+), 354 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/7
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 7
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 6:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3567/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Hello Ian Maxon, Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1380

to look at the new patch set (#6).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
D 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointObject.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
21 files changed, 740 insertions(+), 354 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/6
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Introduce CheckpointManager API

[Ian Maxon (Code Review)]

Ian Maxon has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5: Code-Review+2

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Murtadha Hubail has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5:

(2 comments)

https://asterix-gerrit.ics.uci.edu/#/c/1380/5/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java:

Line 104: result = prime * result + (int) (minMCTFirstLsn ^ 
(minMCTFirstLsn >>> 32));
> What's the 1231 & 1237 about?
This is an auto-generated hashCode method by Eclipse that uses all attributes 
in the class. These two numbers are just prime numbers chosen by Eclipse.


https://asterix-gerrit.ics.uci.edu/#/c/1380/5/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
File 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java:

Line 128: 
> What's this about?
The capture method? It just captures a checkpoint the specified LSN
as follows 1) create checkpoint object. 2) persist it to disk. 3) run a cleanup 
to delete old checkpoints.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Introduce CheckpointManager API

[Ian Maxon (Code Review)]

Ian Maxon has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5: Code-Review+1

(2 comments)

https://asterix-gerrit.ics.uci.edu/#/c/1380/5/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java:

Line 104: result = prime * result + (int) (minMCTFirstLsn ^ 
(minMCTFirstLsn >>> 32));
What's the 1231 & 1237 about?


https://asterix-gerrit.ics.uci.edu/#/c/1380/5/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
File 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java:

Line 128: 
What's this about?


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1302/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1302/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Till Westmann (Code Review)]

Till Westmann has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4:

(1 comment)

https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java:

Line 29: private static final int LOCAL_STORAGE_VERSION = 1;
> ITreeIndexMetaDataFrame.VERSION is Hyracks storage version. It shouldn't be
Good point :)


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Murtadha Hubail has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4:

(3 comments)

https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java:

Line 251: if (!pendingFailbackCompletion) {
> It's not a checkpoint, but it's checkpointy ;)
Done ;)


https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java:

Line 29: private static final int LOCAL_STORAGE_VERSION = 1;
> Why a separate version?
ITreeIndexMetaDataFrame.VERSION is Hyracks storage version. It shouldn't be 
changed just because some Asterix related storage artifact format changed (e.g. 
checkpoint format). Ideally, even the Hyracks storage version should be passed 
from the App running on top of Hyracks. The VERSON attribute below combines 
both the Asterix version as well as the Hyracks storage version to ensure that 
if any of them changed then a version mismatch is encountered.


https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
File 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java:

Line 28: /**
> s/demon/daemon
Done


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1301/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5: Integration-Tests-1

Integration Tests Failed

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1300/ 
: UNSTABLE

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/29/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1300/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 5:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3524/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1380

to look at the new patch set (#5).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
D 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointObject.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
21 files changed, 738 insertions(+), 352 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/5
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Introduce CheckpointManager API

[Ian Maxon (Code Review)]

Ian Maxon has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4:

(3 comments)

https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java:

Line 251: if (!pendingFailbackCompletion) {
It's not a checkpoint, but it's checkpointy ;)


https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java:

Line 29: private static final int LOCAL_STORAGE_VERSION = 1;
Why a separate version?


https://asterix-gerrit.ics.uci.edu/#/c/1380/4/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
File 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java:

Line 28: /**
s/demon/daemon


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Introduce CheckpointManager API

[Till Westmann (Code Review)]

Till Westmann has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4: Code-Review+1

Good for me. Somebody with more area knowledge (Abdullah or Ian or ...?) should 
give the +2.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1282/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1282/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 4:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3501/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1380

to look at the new patch set (#4).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
D 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointObject.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
21 files changed, 738 insertions(+), 352 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 3: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1280/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 3:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1280/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 3:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3499/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1380

to look at the new patch set (#3).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
R 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
20 files changed, 642 insertions(+), 284 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/3
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3498/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Murtadha Hubail has uploaded a new patch set (#2).

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
R 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
20 files changed, 624 insertions(+), 284 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Introduce CheckpointManager API

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: Introduce CheckpointManager API
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3497/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1380
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: Introduce CheckpointManager API

[Murtadha Hubail (Code Review)]

Murtadha Hubail has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1380

Change subject: Introduce CheckpointManager API
..

Introduce CheckpointManager API

This change includes the following:
- s/CheckpointObject/Checkpoint
- Add AsterixDB storage version to checkpoints.
- Prevent any txn log access when a storage version mismatch is detected.
- Introduce CheckpointManager API and CheckpointProperties.
- Properly stop checkpointing thread on instance shutdown.
- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java
R 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/Checkpoint.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/CheckpointProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ICheckpointManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionManager.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionSubsystem.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/AbstractCheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointManagerFactory.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/CheckpointThread.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java
A 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionSubsystem.java
20 files changed, 627 insertions(+), 285 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/80/1380/1

diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
index af50b71..0ba750b 100644
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/AsterixNCAppRuntimeContext.java
@@ -278,7 +278,7 @@
 lccm.register((ILifeCycleComponent) datasetLifecycleManager);
 lccm.register((ILifeCycleComponent) 
txnSubsystem.getTransactionManager());
 lccm.register((ILifeCycleComponent) txnSubsystem.getLockManager());
-
+lccm.register(txnSubsystem.getCheckpointManager());
 }
 
 @Override
diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
index ea1f714..feacb8a 100644
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java
@@ -37,6 +37,7 @@
 import org.apache.asterix.common.config.IAsterixPropertiesProvider;
 import org.apache.asterix.common.config.MessagingProperties;
 import org.apache.asterix.common.replication.IRemoteRecoveryManager;
+import org.apache.asterix.common.transactions.ICheckpointManager;
 import org.apache.asterix.common.transactions.IRecoveryManager;
 import org.apache.asterix.common.transactions.IRecoveryManager.SystemState;
 import org.apache.asterix.common.utils.PrintUtil;
@@ -47,7 +48,6 @@
 import org.apache.asterix.messaging.NCMessageBroker;
 import