Hello Yingyi Bu, Ian Maxon, Jenkins, I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/1813 to look at the new patch set (#17). Change subject: [ASTERIXDB-1946][STO][IDX] Create BTreeIndex for Correlated Datasets ...................................................................... [ASTERIXDB-1946][STO][IDX] Create BTreeIndex for Correlated Datasets Implemented create seconary BTree index for datasets using correlated merge policy. Instead of creating one component for the new index, this change creates one component for each component of the primary index to maintain the correlation. The current implementation assumes when a secondary index is being created, the dataset is locked with no modifications. Change-Id: I2a3435e6720f07bd6a5092d4d9ce42e8d4b7894c --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.5.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.10.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.11.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.5.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.6.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.7.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.8.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.9.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.10.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.11.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.4.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.5.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.6.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.7.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.8.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/open-index-non-enforced/correlated-index-selection/btree-index-01/btree-index-01.9.adm M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.1.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.10.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.2.ddl.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.3.update.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.4.txneu.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.5.ddl.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.6.txnqbc.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.7.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.8.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.9.txnqar.aql A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/create_and_start.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/kill_cc_and_nc.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/stop_and_delete.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/stop_and_start.sh M asterixdb/asterix-installer/src/test/resources/transactionts/testsuite.xml M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryCorrelatedBTreeOperationsHelper.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryCorrelatedTreeIndexOperationsHelper.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AbstractLSMSecondaryIndexCreationNodePushable.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryIndexBulkLoadNodePushable.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryIndexBulkLoadOperatorDescriptor.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryIndexCreationTupleProcessorNodePushable.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryIndexCreationTupleProcessorOperatorDescriptor.java A asterixdb/asterix-runtime/src/test/java/org/apache/asterix/runtime/operators/LSMSecondaryIndexCreationTupleProcessorTest.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/pom.xml A hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeDiskComponentScanOperatorDescriptor.java A hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeDiskComponentScanOperatorNodePushable.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/ExternalBTreeWithBuddy.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTreeDiskComponentScanCursor.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/impls/LSMInvertedIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-rtree/src/main/java/org/apache/hyracks/storage/am/lsm/rtree/impls/LSMRTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-rtree/src/main/java/org/apache/hyracks/storage/am/lsm/rtree/impls/LSMRTreeWithAntiMatterTuples.java 121 files changed, 4,989 insertions(+), 40 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/1813/17 -- To view, visit https://asterix-gerrit.ics.uci.edu/1813 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a3435e6720f07bd6a5092d4d9ce42e8d4b7894c Gerrit-PatchSet: 17 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen <cl...@uci.edu> Gerrit-Reviewer: Ian Maxon <ima...@apache.org> Gerrit-Reviewer: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Gerrit-Reviewer: Jianfeng Jia <jianfeng....@gmail.com> Gerrit-Reviewer: Luo Chen <cl...@uci.edu> Gerrit-Reviewer: Till Westmann <ti...@apache.org> Gerrit-Reviewer: Yingyi Bu <buyin...@gmail.com> Gerrit-Reviewer: abdullah alamoudi <bamou...@gmail.com>