Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[abdullah alamoudi (Code Review)]

abdullah alamoudi has submitted this change and it was merged.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
- This change introduces some improvements to storage
  operations.
- Local RecoveryManager is now extensible.
- Bulk loaders now call the IO callback similar to
  Flushes, making them less special and creating a
  unified lifecycle for adding an index component.
- As a result, The IndexCheckpointManager doesn't need
  to have a special treatment for components loaded
  through the bulk load operation.
- Component Id have been added to the index checkpoint
  files.
- Cleanup for the code of local recovery for failed flush
  operations.
- Ensure that after local recovery of flushes, primary
  and secondary indexes have the same index for mutable
  memory component.
- The use of WAIT logs to ensure in-flight flushes
  are scheduled didn't work as expected. A new log type
  WAIT_FOR_FLUSHES was introduced to acheive the expected
  behavior.
- The local test framework was made Extensible to support
  more use cases.
- Test cases were added for component ids in checkpoint files.
  The following scenarios were covered:
  - Primary and secondary both have values when a flush is
shceduled.
  - Primary have values but not secondary when a flush is
scheduled.
  - Primary is empty and an index is created through bulk
load.
  - Primary has a single component and secondary is created
through bulk load.
  - Primary has multiple components and secondary is created
through bulk load.
- Each primary opTracker now keeps a list of ongoing flushes.
- FlushDataset now waits only for flushes only and
  not all io operations.
- Previously, we had many flushes scheduled on open datasets.
  This was not detected but after this change, a failure
  is thrown in such cases.
- Flush operations dont need to extend the comparable
  interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632
Sonar-Qube: Jenkins 
Tested-by: Jenkins 
Contrib: Jenkins 
Integration-Tests: Jenkins 
Reviewed-by: Murtadha Hubail 
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ABooleanFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ADoubleFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt32FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt64FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ARecordValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AStringFieldValueGenerator.java
R 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/RecordTupleGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt32ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt64ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/IoCallbackFailureTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LogMarkerTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/SearchCursorComponentSwitchTest.java
M 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Murtadha Hubail (Code Review)]

Murtadha Hubail has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19: Code-Review+2

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6420/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19: Contrib+1

Analytics Compatibility Tests Successful
https://goo.gl/WciTKy : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19: Contrib+1

BAD Compatibility Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/3177/ : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

BAD Compatibility Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/3177/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Analytics Compatibility Compilation Successful
https://goo.gl/PpcdAJ : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6420/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/3468/ 
(6/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/7489/ (4/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/4089/ (7/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[abdullah alamoudi (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/2632

to look at the new patch set (#19).

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..

[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
- This change introduces some improvements to storage
  operations.
- Local RecoveryManager is now extensible.
- Bulk loaders now call the IO callback similar to
  Flushes, making them less special and creating a
  unified lifecycle for adding an index component.
- As a result, The IndexCheckpointManager doesn't need
  to have a special treatment for components loaded
  through the bulk load operation.
- Component Id have been added to the index checkpoint
  files.
- Cleanup for the code of local recovery for failed flush
  operations.
- Ensure that after local recovery of flushes, primary
  and secondary indexes have the same index for mutable
  memory component.
- The use of WAIT logs to ensure in-flight flushes
  are scheduled didn't work as expected. A new log type
  WAIT_FOR_FLUSHES was introduced to acheive the expected
  behavior.
- The local test framework was made Extensible to support
  more use cases.
- Test cases were added for component ids in checkpoint files.
  The following scenarios were covered:
  - Primary and secondary both have values when a flush is
shceduled.
  - Primary have values but not secondary when a flush is
scheduled.
  - Primary is empty and an index is created through bulk
load.
  - Primary has a single component and secondary is created
through bulk load.
  - Primary has multiple components and secondary is created
through bulk load.
- Each primary opTracker now keeps a list of ongoing flushes.
- FlushDataset now waits only for flushes only and
  not all io operations.
- Previously, we had many flushes scheduled on open datasets.
  This was not detected but after this change, a failure
  is thrown in such cases.
- Flush operations dont need to extend the comparable
  interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ABooleanFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ADoubleFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt32FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt64FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ARecordValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AStringFieldValueGenerator.java
R 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/RecordTupleGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt32ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt64ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/IoCallbackFailureTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LogMarkerTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/SearchCursorComponentSwitchTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/StorageTestUtils.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestDataset.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TransactionAbortTest.java
M 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-format/3515/ 
(8/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/3771/ 
(2/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/3556/
 (10/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/4274/ (5/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/9019/ (1/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/4009/ 
(12/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-ensure-ancestor/1563/ 
(11/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/3544/ 
(9/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 19:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/3987/
 (3/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 19
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6419/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18: Contrib+1

Analytics Compatibility Tests Successful
https://goo.gl/EunovS : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Analytics Compatibility Compilation Successful
https://goo.gl/1o7eXg : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/4273/ (12/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[abdullah alamoudi (Code Review)]

abdullah alamoudi has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

(26 comments)

https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java:

PS17, Line 122: String validComponentTimestamp;
> revert
Done


Line 221: 
> public methods up
Done


Line 223: public void setLastComponentId(long componentId) throws 
HyracksDataException {
> synchronized
Done


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java:

PS17, Line 64: public
> Move public method up
Done


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/PrimaryIndexOperationTracker.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/PrimaryIndexOperationTracker.java:

PS17, Line 61:  M
> final
Done


PS17, Line 61: ongoingFlushes
> scheduledFlushes
Done


PS17, Line 211: getOngoingFlushes
> getScheduledFlushes
Done


PS17, Line 211: void
> Return a list or unmodifiable list.
Done


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/ioopcallbacks/LSMIOOperationCallback.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/ioopcallbacks/LSMIOOperationCallback.java:

PS17, Line 123: // Get component id of the last disk component
  : LSMComponentId mostRecentComponentId =
  : 
mostRecent(operation.getAccessor().getOpContext().getComponentsToBeMerged());
  : // Update the checkpoint file
  : FileReference target = operation.getTarget();
  : final ResourceReference ref = 
ResourceReference.of(target.getAbsolutePath());
  : 
indexCheckpointManagerProvider.get(ref).setLastComponentId(mostRecentComponentId.getMaxId());
> extract with proper name
Done


PS17, Line 136: FileReference target = operation.getTarget();
  : Map map = operation.getParameters();
  : final Long lsn =
  : operation.getIOOpertionType() == 
LSMIOOperationType.FLUSH ? (Long) map.get(KEY_FLUSH_LOG_LSN) : 0L;
  : final LSMComponentId id = (LSMComponentId) 
map.get(KEY_FLUSHED_COMPONENT_ID);
  : final ResourceReference ref = 
ResourceReference.of(target.getAbsolutePath());
  : final String componentEndTime = 
AbstractLSMIndexFileManager.getComponentEndTime(ref.getName());
  : 
indexCheckpointManagerProvider.get(ref).flushed(componentEndTime, lsn, 
id.getMaxId())
> extract with proper name
Done


PS17, Line 147: mostRecent
> getMostRecentComponentId
Done


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IIndexCheckpointManagerProvider.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IIndexCheckpointManagerProvider.java:

PS17, Line 43: /**
 :  * Get the path of the index containing the passed reference
 :  *
 :  * @param ref
 :  * @return
 :  * @throws HyracksDataException
 :  */
 : Path getIndexPath(ResourceReference ref) throws 
HyracksDataException;
> Move this to a better place (e.g. PersistentLocalResourceRepository)
Moved to StoragePathUtil


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexCheckpoint.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexCheckpoint.java:

PS17, Line 36: EMPTY_INDEX_LAST_COMPONENT_ID
> This doesn't belong here. Maybe move it to LSMComponentId?
Done


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogType.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogType.java:

PS17, Line 41: WAIT
> Fix this
Done


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/MarkComponentValidTask.java
File 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/MarkComponentValidTask.java:

PS17, Line 47: masterLsn < 
> A better way to do this is to define a constant called 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/7488/ (9/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-ensure-ancestor/1562/ 
(5/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/4088/ (11/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/3543/ 
(3/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6419/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/4008/ 
(4/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/3555/
 (7/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[abdullah alamoudi (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/2632

to look at the new patch set (#18).

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..

[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
- This change introduces some improvements to storage
  operations.
- Local RecoveryManager is now extensible.
- Bulk loaders now call the IO callback similar to
  Flushes, making them less special and creating a
  unified lifecycle for adding an index component.
- As a result, The IndexCheckpointManager doesn't need
  to have a special treatment for components loaded
  through the bulk load operation.
- Component Id have been added to the index checkpoint
  files.
- Cleanup for the code of local recovery for failed flush
  operations.
- Ensure that after local recovery of flushes, primary
  and secondary indexes have the same index for mutable
  memory component.
- The use of WAIT logs to ensure in-flight flushes
  are scheduled didn't work as expected. A new log type
  WAIT_FOR_FLUSHES was introduced to acheive the expected
  behavior.
- The local test framework was made Extensible to support
  more use cases.
- Test cases were added for component ids in checkpoint files.
  The following scenarios were covered:
  - Primary and secondary both have values when a flush is
shceduled.
  - Primary have values but not secondary when a flush is
scheduled.
  - Primary is empty and an index is created through bulk
load.
  - Primary has a single component and secondary is created
through bulk load.
  - Primary has multiple components and secondary is created
through bulk load.
- Each primary opTracker now keeps a list of ongoing flushes.
- FlushDataset now waits only for flushes only and
  not all io operations.
- Previously, we had many flushes scheduled on open datasets.
  This was not detected but after this change, a failure
  is thrown in such cases.
- Flush operations dont need to extend the comparable
  interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ABooleanFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ADoubleFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt32FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt64FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ARecordValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AStringFieldValueGenerator.java
R 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/RecordTupleGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt32ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt64ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/IoCallbackFailureTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LogMarkerTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/SearchCursorComponentSwitchTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/StorageTestUtils.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestDataset.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TransactionAbortTest.java
M 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/9018/ (1/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/3467/ 
(8/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/3986/
 (6/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-format/3514/ 
(2/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 18:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/3770/ 
(10/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 18
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Murtadha Hubail (Code Review)]

Murtadha Hubail has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

(26 comments)

https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java:

PS17, Line 122: String validComponentTimestamp;
revert


Line 221: 
public methods up


Line 223: public void setLastComponentId(long componentId) throws 
HyracksDataException {
synchronized


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java:

PS17, Line 64: public
Move public method up


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/PrimaryIndexOperationTracker.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/PrimaryIndexOperationTracker.java:

PS17, Line 61: ongoingFlushes
scheduledFlushes


PS17, Line 61:  M
final


PS17, Line 211: void
Return a list or unmodifiable list.


PS17, Line 211: getOngoingFlushes
getScheduledFlushes


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/ioopcallbacks/LSMIOOperationCallback.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/ioopcallbacks/LSMIOOperationCallback.java:

PS17, Line 123: // Get component id of the last disk component
  : LSMComponentId mostRecentComponentId =
  : 
mostRecent(operation.getAccessor().getOpContext().getComponentsToBeMerged());
  : // Update the checkpoint file
  : FileReference target = operation.getTarget();
  : final ResourceReference ref = 
ResourceReference.of(target.getAbsolutePath());
  : 
indexCheckpointManagerProvider.get(ref).setLastComponentId(mostRecentComponentId.getMaxId());
extract with proper name


PS17, Line 136: FileReference target = operation.getTarget();
  : Map map = operation.getParameters();
  : final Long lsn =
  : operation.getIOOpertionType() == 
LSMIOOperationType.FLUSH ? (Long) map.get(KEY_FLUSH_LOG_LSN) : 0L;
  : final LSMComponentId id = (LSMComponentId) 
map.get(KEY_FLUSHED_COMPONENT_ID);
  : final ResourceReference ref = 
ResourceReference.of(target.getAbsolutePath());
  : final String componentEndTime = 
AbstractLSMIndexFileManager.getComponentEndTime(ref.getName());
  : 
indexCheckpointManagerProvider.get(ref).flushed(componentEndTime, lsn, 
id.getMaxId())
extract with proper name


PS17, Line 147: mostRecent
getMostRecentComponentId


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IIndexCheckpointManagerProvider.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IIndexCheckpointManagerProvider.java:

PS17, Line 43: /**
 :  * Get the path of the index containing the passed reference
 :  *
 :  * @param ref
 :  * @return
 :  * @throws HyracksDataException
 :  */
 : Path getIndexPath(ResourceReference ref) throws 
HyracksDataException;
Move this to a better place (e.g. PersistentLocalResourceRepository)


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexCheckpoint.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexCheckpoint.java:

PS17, Line 36: EMPTY_INDEX_LAST_COMPONENT_ID
This doesn't belong here. Maybe move it to LSMComponentId?


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogType.java
File 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogType.java:

PS17, Line 41: WAIT
Fix this


https://asterix-gerrit.ics.uci.edu/#/c/2632/17/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/MarkComponentValidTask.java
File 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/MarkComponentValidTask.java:

PS17, Line 47: masterLsn < 
A better way to do this is to define a constant called MERGE_LSN.
Either remove the comment or link to LSMIOOperationCallback.INVALID_LSN.


PS17, Line 62: 0
Make this > 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17: Contrib+1

Analytics Compatibility Tests Successful
https://goo.gl/85D28C : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6392/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16: Contrib+1

Analytics Compatibility Tests Successful
https://goo.gl/h2u7gU : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17: Contrib+1

BAD Compatibility Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/3161/ : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

BAD Compatibility Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/3161/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Analytics Compatibility Compilation Successful
https://goo.gl/SGzzA6 : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Analytics Compatibility Compilation Successful
https://goo.gl/yb6tGi : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6392/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-ensure-ancestor/1531/ 
(12/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/4242/ (10/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/7461/ (9/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/4057/ (11/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/3955/
 (4/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/3512/ 
(6/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/3524/
 (3/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/3436/ 
(5/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/3739/ 
(8/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[abdullah alamoudi (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/2632

to look at the new patch set (#17).

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..

[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
- This change introduces some improvements to storage
  operations.
- Local RecoveryManager is now extensible.
- Bulk loaders now call the IO callback similar to
  Flushes, making them less special and creating a
  unified lifecycle for adding an index component.
- As a result, The IndexCheckpointManager doesn't need
  to have a special treatment for components loaded
  through the bulk load operation.
- Component Id have been added to the index checkpoint
  files.
- Cleanup for the code of local recovery for failed flush
  operations.
- Ensure that after local recovery of flushes, primary
  and secondary indexes have the same index for mutable
  memory component.
- The use of WAIT logs to ensure in-flight flushes
  are scheduled didn't work as expected. A new log type
  WAIT_FOR_FLUSHES was introduced to acheive the expected
  behavior.
- The local test framework was made Extensible to support
  more use cases.
- Test cases were added for component ids in checkpoint files.
  The following scenarios were covered:
  - Primary and secondary both have values when a flush is
shceduled.
  - Primary have values but not secondary when a flush is
scheduled.
  - Primary is empty and an index is created through bulk
load.
  - Primary has a single component and secondary is created
through bulk load.
  - Primary has multiple components and secondary is created
through bulk load.
- Each primary opTracker now keeps a list of ongoing flushes.
- FlushDataset now waits only for flushes only and
  not all io operations.
- Previously, we had many flushes scheduled on open datasets.
  This was not detected but after this change, a failure
  is thrown in such cases.
- Flush operations dont need to extend the comparable
  interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ABooleanFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ADoubleFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt32FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt64FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ARecordValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AStringFieldValueGenerator.java
R 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/RecordTupleGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt32ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt64ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/IoCallbackFailureTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LogMarkerTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/SearchCursorComponentSwitchTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/StorageTestUtils.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestDataset.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TransactionAbortTest.java
M 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-format/3483/ 
(7/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/3977/ 
(2/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/8987/ (1/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[abdullah alamoudi (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/2632

to look at the new patch set (#16).

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..

[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
- This change introduces some improvements to storage
  operations.
- Bulk loaders now call the IO callback similar to
  Flushes, making them less special and creating a
  unified lifecycle for adding an index component.
- As a result, The IndexCheckpointManager doesn't need
  to have a special treatment for components loaded
  through the bulk load operation.
- Component Id have been added to the index checkpoint
  files.
- Cleanup for the code of local recovery for failed flush
  operations.
- Ensure that after local recovery of flushes, primary
  and secondary indexes have the same index for mutable
  memory component.
- The use of WAIT logs to ensure in-flight flushes
  are scheduled didn't work as expected. A new log type
  WAIT_FOR_FLUSHES was introduced to acheive the expected
  behavior.
- The local test framework was made Extensible to support
  more use cases.
- Test cases were added for component ids in checkpoint files.
  The following scenarios were covered:
  - Primary and secondary both have values when a flush is
shceduled.
  - Primary have values but not secondary when a flush is
scheduled.
  - Primary is empty and an index is created through bulk
load.
  - Primary has a single component and secondary is created
through bulk load.
  - Primary has multiple components and secondary is created
through bulk load.
- Each primary opTracker now keeps a list of ongoing flushes.
- FlushDataset now waits only for flushes only and
  not all io operations.
- Previously, we had many flushes scheduled on open datasets.
  This was not detected but after this change, a failure
  is thrown in such cases.
- Flush operations dont need to extend the comparable
  interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/IndexCheckpointManagerProvider.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ABooleanFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ADoubleFieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt32FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AInt64FieldValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/ARecordValueGenerator.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/AStringFieldValueGenerator.java
R 
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/data/gen/RecordTupleGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt32ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/AInt64ValueGenerator.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/CheckpointInSecondaryIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/IoCallbackFailureTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LSMFlushRecoveryTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LogMarkerTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/MultiPartitionLSMIndexTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/SearchCursorComponentSwitchTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/StorageTestUtils.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestDataset.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TransactionAbortTest.java
M 

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/3523/
 (6/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-ensure-ancestor/1530/ 
(10/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/4241/ (8/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/3511/ 
(12/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/7460/ (2/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/8986/ (1/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/3954/
 (4/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/3976/ 
(3/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/3435/ 
(5/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-format/3482/ 
(11/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/3738/ 
(7/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Misc Storage Fixes and Improvements

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Misc Storage Fixes and Improvements
..


Patch Set 16:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/4056/ (9/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2632
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No