Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.
Hello Anon. E. Moose #1000171, Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/1076 to look at the new patch set (#79). Change subject: Applied the multiway prefix-based fuzzyjoin. .. Applied the multiway prefix-based fuzzyjoin. - Enable the fuzzyjoin rule using prefix-based similarity join for RS cases. - Introduce eight existing rules in FuzzyJoinRuleCollections after applied the fuzzyjoin rule. - Add ExtractCommonExpressionRule to extract the common expressions in the star-like multiple fuzzyjoin substitutions. - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated subplan from the similarity function-derived substitution into join in case of nested schemas. - Use similarity-jaccard-prefix to enable the pp+ join strategy. - Use right side to build the heavy hash join on the prefix tokens from both sides. - Add RemoveAssign/Variables/AggRules to iteratively remove unused assign/vars once FuzzyJoinRule is applied in each round. - Add three new optimization Cases for multi-fuzzyjoin. - link-like multiway fuzzyjoins - star-like multiway fuzzyjoins - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins. - Add a running Case where another fuzzyjoin is regarded as select over an existing prefix-based fuzzyjoin. - Change the inverted-index-based fuzzyjoin onto the prefix-based join due to efficiency considerations. - Increase the ManagixExecutionIT nc memory configuration so as to enable multi-way fuzzy join to pass the capacity check. Change-Id: I8736f104905eeda763d39709e002c2b9629278cc --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt A asterixdb/asterix-app/data/pub-small/csxauthors.adm A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt A asterixdb/asterix-app/data/pub-small/dblpauthors.adm M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt A asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql A asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql A asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql A asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql A asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql A asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql A asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan A asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan A asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan A asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_
Change in asterixdb[master]: [NO ISSUE][RT][IDX] Simplify index.createAccessor()
Jenkins has posted comments on this change. Change subject: [NO ISSUE][RT][IDX] Simplify index.createAccessor() .. Patch Set 5: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5457/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2120 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iae015c342e830c81d666428447b595280139740e Gerrit-PatchSet: 5 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][RT][IDX] Simplify index.createAccessor()
Jenkins has posted comments on this change. Change subject: [NO ISSUE][RT][IDX] Simplify index.createAccessor() .. Patch Set 5: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1410/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2120 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iae015c342e830c81d666428447b595280139740e Gerrit-PatchSet: 5 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][RT][IDX] Simplify index.createAccessor()
Jenkins has posted comments on this change. Change subject: [NO ISSUE][RT][IDX] Simplify index.createAccessor() .. Patch Set 5: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6953/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2120 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iae015c342e830c81d666428447b595280139740e Gerrit-PatchSet: 5 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][RT][IDX] Simplify index.createAccessor()
Jenkins has posted comments on this change. Change subject: [NO ISSUE][RT][IDX] Simplify index.createAccessor() .. Patch Set 5: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/2120 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iae015c342e830c81d666428447b595280139740e Gerrit-PatchSet: 5 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][RT][IDX] Simplify index.createAccessor()
Jenkins has posted comments on this change. Change subject: [NO ISSUE][RT][IDX] Simplify index.createAccessor() .. Patch Set 5: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2016/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2120 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iae015c342e830c81d666428447b595280139740e Gerrit-PatchSet: 5 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1735/ (7/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][RT][IDX] Simplify index.createAccessor()
Jenkins has posted comments on this change. Change subject: [NO ISSUE][RT][IDX] Simplify index.createAccessor() .. Patch Set 5: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1402/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2120 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iae015c342e830c81d666428447b595280139740e Gerrit-PatchSet: 5 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/2099/ (10/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
[jira] [Closed] (ASTERIXDB-2145) Recovery process fails on 100 datasets
[ https://issues.apache.org/jira/browse/ASTERIXDB-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murtadha Hubail closed ASTERIXDB-2145. -- Resolution: Duplicate > Recovery process fails on 100 datasets > -- > > Key: ASTERIXDB-2145 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2145 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Assignee: Ian Maxon >Priority: Major > > On the Cloudberry DB, currently, there are 112 datasets on a dataverse. When > restarting that instance, the NC showed the following error and stopped. > java.lang.IllegalStateException: Failed to redo > at org.apache.asterix.app.nc.RecoveryManager.redo(RecoveryManager.java:712) > at > org.apache.asterix.app.nc.RecoveryManager.startRecoveryRedoPhase(RecoveryManager.java:378) > at > org.apache.asterix.app.nc.RecoveryManager.replayPartitionsLogs(RecoveryManager.java:187) > at > org.apache.asterix.app.nc.RecoveryManager.startLocalRecovery(RecoveryManager.java:179) > at > org.apache.asterix.app.nc.task.LocalRecoveryTask.perform(LocalRecoveryTask.java:43) > at > org.apache.asterix.app.replication.message.StartupTaskResponseMessage.handle(StartupTaskResponseMessage.java:56) > at > org.apache.asterix.messaging.NCMessageBroker.receivedMessage(NCMessageBroker.java:92) > at > org.apache.hyracks.control.nc.work.ApplicationMessageWork.run(ApplicationMessageWork.java:51) > at > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127) > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: > Cannot allocate dataset 191 memory since memory budget would be > exceeded. > at > org.apache.asterix.common.context.DatasetLifecycleManager.allocateMemory(DatasetLifecycleManager.java:568) > at > org.apache.hyracks.storage.common.buffercache.ResourceHeapBufferAllocator.reserveAllocation(ResourceHeapBufferAllocator.java:53) > at > org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.open(VirtualBufferCache.java:307) > at > org.apache.hyracks.storage.am.lsm.common.impls.MultitenantVirtualBufferCache.open(MultitenantVirtualBufferCache.java:119) > at > org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.allocateMemoryComponent(LSMBTree.java:611) > at > org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.allocateMemoryComponents(AbstractLSMIndex.java:389) > at > org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:421) > at > org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.forceModify(LSMHarness.java:368) > at > org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.forceUpsert(LSMTreeIndexAccessor.java:181) > at org.apache.asterix.app.nc.RecoveryManager.redo(RecoveryManager.java:707) > ... 8 more > So, I increased the storage.memorycomponent.globalbudget parameter from 3GB > to 5GB. Still, the NC showed the following error and the recovery process > could not finish. > ... similar log records ... > Oct 25, 2017 9:33:44 AM > org.apache.asterix.transaction.management.resource.PersistentLocalResourceRepository > loadDataverse > INFO: Loading dataverse:berry > Oct 25, 2017 9:33:44 AM > org.apache.asterix.transaction.management.resource.PersistentLocalResourceRepository > loadIndex > INFO: Loading index:meta_idx_meta > Oct 25, 2017 9:33:44 AM > org.apache.asterix.transaction.management.resource.PersistentLocalResourceRepository > loadIndex > INFO: Resource loaded 161:storage/partition_1/berry/meta_idx_meta > Oct 25, 2017 9:34:09 AM org.apache.hyracks.util.ExitUtil$ExitThread run > INFO: JVM exiting with status 2; bye! > So, I checked the parameter information page and found that the default > parameter for storage.memorycomponent.numpages is 1/16 of the global > component budget. Therefore, I decreased this parameter to increase the > number of datasets in memory. And the instance was finally able to start. So, > it seems that the recovery process tries to load and keep all datasets into > memory and this needs to be checked. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (ASTERIXDB-2081) Failed to restart after hit an OOM issue
[
https://issues.apache.org/jira/browse/ASTERIXDB-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murtadha Hubail resolved ASTERIXDB-2081.
Resolution: Fixed
> Failed to restart after hit an OOM issue
>
>
> Key: ASTERIXDB-2081
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2081
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: STO - Storage
> Environment: master
>Reporter: Jianfeng Jia
>Assignee: Murtadha Hubail
>Priority: Major
>
> One of the node was failed due to the OOM error. Then when we try to restart
> the service, the node couldn't be recovered and the logs is shown as below:
> {code}
> WARNING: Error in application message delivery!
> java.lang.IllegalStateException: Failed to redo
> at
> org.apache.asterix.app.nc.RecoveryManager.redo(RecoveryManager.java:712)
> at
> org.apache.asterix.app.nc.RecoveryManager.startRecoveryRedoPhase(RecoveryManager.java:378)
> at
> org.apache.asterix.app.nc.RecoveryManager.replayPartitionsLogs(RecoveryManager.java:187)
> at
> org.apache.asterix.app.nc.RecoveryManager.startLocalRecovery(RecoveryManager.java:179)
> at
> org.apache.asterix.app.nc.task.LocalRecoveryTask.perform(LocalRecoveryTask.java:43)
> at
> org.apache.asterix.app.replication.message.StartupTaskResponseMessage.handle(StartupTaskResponseMessage.java:53)
> at
> org.apache.asterix.messaging.NCMessageBroker.receivedMessage(NCMessageBroker.java:92)
> at
> org.apache.hyracks.control.nc.work.ApplicationMessageWork.run(ApplicationMessageWork.java:54)
> at
> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Cannot
> allocate dataset 245 memory since memory budget would be exceeded.
> at
> org.apache.asterix.common.context.DatasetLifecycleManager.allocateMemory(DatasetLifecycleManager.java:566)
> at
> org.apache.hyracks.storage.common.buffercache.ResourceHeapBufferAllocator.reserveAllocation(ResourceHeapBufferAllocator.java:53)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.open(VirtualBufferCache.java:307)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.MultitenantVirtualBufferCache.open(MultitenantVirtualBufferCache.java:119)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.allocateMemoryComponent(LSMBTree.java:602)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.allocateMemoryComponents(AbstractLSMIndex.java:386)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:417)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.forceModify(LSMHarness.java:364)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.forceUpsert(LSMTreeIndexAccessor.java:181)
> at
> org.apache.asterix.app.nc.RecoveryManager.redo(RecoveryManager.java:707)
> ... 8 more
> Sep 05, 2017 3:37:46 PM
> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> WARNING: Exception while executing ApplicationMessage: nodeID: 4
> java.lang.RuntimeException: java.lang.IllegalStateException: Failed to redo
> at
> org.apache.hyracks.control.nc.work.ApplicationMessageWork.run(ApplicationMessageWork.java:60)
> at
> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127)
> Caused by: java.lang.IllegalStateException: Failed to redo
> at
> org.apache.asterix.app.nc.RecoveryManager.redo(RecoveryManager.java:712)
> at
> org.apache.asterix.app.nc.RecoveryManager.startRecoveryRedoPhase(RecoveryManager.java:378)
> at
> org.apache.asterix.app.nc.RecoveryManager.replayPartitionsLogs(RecoveryManager.java:187)
> at
> org.apache.asterix.app.nc.RecoveryManager.startLocalRecovery(RecoveryManager.java:179)
> at
> org.apache.asterix.app.nc.task.LocalRecoveryTask.perform(LocalRecoveryTask.java:43)
> at
> org.apache.asterix.app.replication.message.StartupTaskResponseMessage.handle(StartupTaskResponseMessage.java:53)
> at
> org.apache.asterix.messaging.NCMessageBroker.receivedMessage(NCMessageBroker.java:92)
> at
> org.apache.hyracks.control.nc.work.ApplicationMessageWork.run(ApplicationMessageWork.java:54)
> ... 1 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Cannot
> allocate dataset 245 memory since memory budget would be exceeded.
> at
> org.apache.asterix.common.context.DatasetLifecycleManager.allocateMemory(DatasetLifecycleManager.java:566)
> at
> org.apache.hyracks.storage.common.buffercache.ResourceHeapBufferAllocator.reserveAllocation(ResourceHeapBufferAllocator.java:53)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.open(Vi
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1401/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5456/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2015/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1408/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1409/ (6/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5455/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6952/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1409/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6951/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2122 to look at the new patch set (#2). Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion - user model changes: no - storage format changes: no - interface changes: yes Added ITransactionOperationTracker for txn specific op tracker. Added complete to ITransactionManager to release all resources held by the txn upon its completion. Details: Currently, a dataset could be evicted/dropped while a transaction waiting for its completion (commit/rollback). This change prevents that by incrementing the reference counter of the datasets of all indexes registered in the transaction. Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 --- M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/BaseOperationTracker.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/ITransactionOperationTracker.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionContext.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java 5 files changed, 83 insertions(+), 21 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/22/2122/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1400/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. Patch Set 4: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2014/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF...
Hello Anon. E. Moose #1000171, Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2114 to look at the new patch set (#4). Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation .. [ASTERIXDB-2152][FUN] Enable utilizing multiple cores in UDF evaluation - user model changes: no - storage format changes: no - interface changes: no Details: This change enables specifying runtime parallelism in compiler parameter for UDF evalution in feed pipeline. By specifying the parallelim level in the start feed query, the UDF will be evaluated at the specified parallel level (limited by the number of cores on each node). An example is as follow, SET `compiler.parallelism` "7"; start feed TweetFeed; Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/feeds/feed-with-external-function/feed-with-external-function.3.update.sqlpp M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/FeedDataSource.java 2 files changed, 4 insertions(+), 2 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/14/2114/4 -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Murtadha Hubail has submitted this change and it was merged.
Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
..
[ASTERIXDB-1954][STO][RT] Add Index Drop Options
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- Add options to allow drop index to ignore
index does not exist and retry on index in-use.
- Add test case for new index drop options.
Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2121
Reviewed-by: Michael Blow
Integration-Tests: Michael Blow
Tested-by: Michael Blow
---
M
asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java
A
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/IndexDropOperatorNodePushableTest.java
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryTreeIndexOperationsHelper.java
M
hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java
M
hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties
M
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDataflowHelper.java
M
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDropOperatorDescriptor.java
M
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDropOperatorNodePushable.java
M
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexLifecycleManager.java
13 files changed, 309 insertions(+), 31 deletions(-)
Approvals:
Michael Blow: Looks good to me, approved; Verified; Verified
diff --git
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java
index ceaf4cf..9e34d70 100644
---
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java
+++
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java
@@ -19,8 +19,10 @@
package org.apache.asterix.utils;
import static org.apache.asterix.app.translator.QueryTranslator.abort;
+import static
org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorDescriptor.DropOption;
import java.util.ArrayList;
+import java.util.EnumSet;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@@ -338,7 +340,8 @@
List jobs = new ArrayList<>();
List indexes =
metadataProvider.getDatasetIndexes(dataset.getDataverseName(),
dataset.getDatasetName());
for (Index index : indexes) {
-jobs.add(IndexUtil.buildDropIndexJobSpec(index, metadataProvider,
dataset, true));
+jobs.add(IndexUtil.buildDropIndexJobSpec(index, metadataProvider,
dataset,
+EnumSet.of(DropOption.IF_EXISTS,
DropOption.WAIT_ON_IN_USE)));
}
for (JobSpecification jobSpec : jobs) {
JobUtils.runJob(hcc, jobSpec, true);
diff --git
a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/IndexDropOperatorNodePushableTest.java
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/IndexDropOperatorNodePushableTest.java
new file mode 100644
index 000..2bac49e
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/IndexDropOperatorNodePushableTest.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.test.storage;
+
+import static
org.apache.hyracks.storage.am.common.dataflow.IndexDropOperator
[jira] [Commented] (ASTERIXDB-1954) RebalanceWithCancellation Test Failed
[
https://issues.apache.org/jira/browse/ASTERIXDB-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237893#comment-16237893
]
ASF subversion and git services commented on ASTERIXDB-1954:
Commit 52e5134d1ed2b16af367eb1283af65e38deb0a07 in asterixdb's branch
refs/heads/master from [~mhubail]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=52e5134 ]
[ASTERIXDB-1954][STO][RT] Add Index Drop Options
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- Add options to allow drop index to ignore
index does not exist and retry on index in-use.
- Add test case for new index drop options.
Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2121
Reviewed-by: Michael Blow
Integration-Tests: Michael Blow
Tested-by: Michael Blow
> RebalanceWithCancellation Test Failed
> -
>
> Key: ASTERIXDB-1954
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1954
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB, STO - Storage
>Reporter: Chen Luo
>Assignee: Murtadha Hubail
>Priority: Major
>
> The RebalanceWithCancellation Test failed with recent changes. The build is
> at:
> https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/924/
> The stacktrace is:
> {code}
> org.apache.hyracks.api.exceptions.HyracksDataException: HYR0081: File
> /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc2/iodevice0/storage/partition_2/tpch/2/LineItem_idx_LineItem_virtual_0
> is already mapped
> at
> org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49)
> at
> org.apache.hyracks.storage.common.file.FileMapManager.registerFile(FileMapManager.java:76)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.createFile(VirtualBufferCache.java:79)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.MultitenantVirtualBufferCache.createFile(MultitenantVirtualBufferCache.java:49)
> at
> org.apache.hyracks.storage.am.common.impls.AbstractTreeIndex.create(AbstractTreeIndex.java:83)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.allocateMemoryComponent(LSMBTree.java:602)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.allocateMemoryComponents(AbstractLSMIndex.java:386)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.enter(LSMHarness.java:623)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.batchOperate(LSMHarness.java:646)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.batchOperate(LSMTreeIndexAccessor.java:214)
> at
> org.apache.asterix.runtime.operators.LSMPrimaryUpsertOperatorNodePushable.nextFrame(LSMPrimaryUpsertOperatorNodePushable.java:280)
> at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:376)
> at org.apache.hyracks.control.nc.Task.run(Task.java:316)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Jun 24, 2017 6:18:34 PM
> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> INFO: Executing: NotifyTaskFailure
> Jun 24, 2017 6:18:34 PM org.apache.hyracks.control.nc.Task run
> WARNING: Task TAID:TID:ANID:ODID:2:0:3:0 failed with exception
> org.apache.hyracks.api.exceptions.HyracksDataException: HYR0081: File
> /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc2/iodevice1/storage/partition_3/tpch/2/LineItem_idx_LineItem_virtual_0
> is already mapped
> at
> org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49)
> at
> org.apache.hyracks.storage.common.file.FileMapManager.registerFile(FileMapManager.java:76)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.createFile(VirtualBufferCache.java:79)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.MultitenantVirtualBufferCache.createFile(MultitenantVirtualBufferCache.java:49)
> at
> org.apache.hyracks.storage.am.common.impls.AbstractTreeIndex.create(AbstractTreeIndex.java:83)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.allocateMemoryComponent(LSMBTree.java:602)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.allocateMemoryComponents(AbstractLSMIndex.java:386)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.enter(LSMHarness.java:623)
> at
> org.apach
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1734/ (7/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/2290/ (9/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: BAD Compatibility Tests Started https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/2059/ -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Michael Blow has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Code-Review+2 Integration-Tests+1 Verified+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1408/ (6/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5454/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Hello Anon. E. Moose #1000171, Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2121 to look at the new patch set (#3). Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. [ASTERIXDB-1954][STO][RT] Add Index Drop Options - user model changes: no - storage format changes: no - interface changes: no Details: - Add options to allow drop index to ignore index does not exist and retry on index in-use. - Add test case for new index drop options. Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/IndexDropOperatorNodePushableTest.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryTreeIndexOperationsHelper.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java M hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDataflowHelper.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDropOperatorDescriptor.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDropOperatorNodePushable.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexLifecycleManager.java 13 files changed, 309 insertions(+), 31 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/21/2121/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1407/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6950/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2013/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1399/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2121 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/2074/ (8/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1406/ (6/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2012/ (7/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1407/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1733/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6949/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5453/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1398/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2122 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Com...
Murtadha Hubail has uploaded a new change for review.
https://asterix-gerrit.ics.uci.edu/2122
Change subject: [ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn
Completion
..
[ASTERIXDB-2130][TX] Avoid Evicting Datasets Pending Txn Completion
- user model changes: no
- storage format changes: no
- interface changes: yes
Added ITransactionOperationTracker for txn specific op tracker.
Added complete to ITransactionManager to release all resources
held by the txn upon its completion.
Details:
Currently, a dataset could be evicted/dropped while a transaction
waiting for its completion (commit/rollback). This change prevents
that by incrementing the reference counter of the datasets of all
indexes registered in the transaction.
Change-Id: I82b614a938f24f3199fd327502ed422ce9d3b9b7
---
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/BaseOperationTracker.java
A
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/ITransactionOperationTracker.java
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionContext.java
M
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java
M
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionManager.java
5 files changed, 82 insertions(+), 19 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/22/2122/1
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/BaseOperationTracker.java
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/BaseOperationTracker.java
index e74600e..2068192 100644
---
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/BaseOperationTracker.java
+++
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/BaseOperationTracker.java
@@ -20,12 +20,11 @@
import org.apache.hyracks.api.exceptions.HyracksDataException;
import org.apache.hyracks.storage.am.lsm.common.api.ILSMIndex;
-import org.apache.hyracks.storage.am.lsm.common.api.ILSMOperationTracker;
import org.apache.hyracks.storage.am.lsm.common.api.LSMOperationType;
import org.apache.hyracks.storage.common.IModificationOperationCallback;
import org.apache.hyracks.storage.common.ISearchOperationCallback;
-public class BaseOperationTracker implements ILSMOperationTracker {
+public class BaseOperationTracker implements ITransactionOperationTracker {
protected final int datasetID;
protected final DatasetInfo dsInfo;
@@ -62,4 +61,14 @@
public void exclusiveJobCommitted() throws HyracksDataException {
}
+
+@Override
+public void beforeTransaction() {
+dsInfo.touch();
+}
+
+@Override
+public void afterTransaction() {
+dsInfo.untouch();
+}
}
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/ITransactionOperationTracker.java
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/ITransactionOperationTracker.java
new file mode 100644
index 000..e6aeec5
--- /dev/null
+++
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/ITransactionOperationTracker.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.common.context;
+
+import org.apache.hyracks.storage.am.lsm.common.api.ILSMOperationTracker;
+
+public interface ITransactionOperationTracker extends ILSMOperationTracker {
+
+/**
+ * Called before a transaction performs any operations on
+ * {@link org.apache.hyracks.storage.am.lsm.common.api.ILSMIndex}
+ */
+void beforeTransaction();
+
+/**
+ * Called after a transaction completes its operations on
+ * {@link org.apache.hyracks.storage.am.lsm.common.api.ILSMIndex}
+ */
+void afterTransaction();
+}
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/ITransactionContext.java
b/asterixdb/asterix-common/src/main/java/org/apache/ast
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1732/ (7/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5452/ (6/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1406/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2011/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2036 to look at the new patch set (#21). Change subject: [WIP] Change Query Plan to Use Component ID .. [WIP] Change Query Plan to Use Component ID Depend on patch: https://asterix-gerrit.ics.uci.edu/#/c/2025/ Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/BTreeSearchPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/InvertedIndexPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/RTreeSearchPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceLSMComponentFilterRule.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceLSMComponentIdRule.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java M asterixdb/asterix-app/src/test/resources/optimizerts/results/filter/btree-btree-search.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/filter/inverted-btree-search.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/filter/multi-index-btree-search.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/filter/rtree-btree-search.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/multi-indexes/btree-rtree-ngram-intersect-with-filter.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/udfs/query-ASTERIXDB-1029.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/udfs/query-ASTERIXDB-1029_2.plan A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-filter/tinysocial-intersect.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-filter/tinysocial-intersect.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-filter/tinysocial-intersect.3.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-nodegroup/intersection-with-nodegroup.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-nodegroup/intersection-with-nodegroup.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-nodegroup/intersection-with-nodegroup.3.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection-with-nodegroup/intersection-with-nodegroup.4.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection/tinysocial-intersect.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection/tinysocial-intersect.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/index-selection/intersection/tinysocial-intersect.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset_with_index/single_dataset_with_index.13.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset_with_index/single_dataset_with_index.8.adm M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedPrefixMergePolicy.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/ioopcallbacks/AbstractLSMIOOperationCallback.java M asterixdb/asterix-common/src/test/java/org/apache/asterix/test/context/CorrelatedPrefixMergePolicyTest.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalBTreeSearchOperatorDescriptor.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalBTreeSearchOperatorNodePushable.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalRTreeSearchOperatorDescriptor.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/ExternalRTreeSearchOperatorNodePushable.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/DataSource.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/DatasetDataSource.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/FeedDataSource.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/LoadableDataSource.java M asterixdb/asterix-metadata/src/
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1405/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6948/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Change Query Plan to Use Component ID
Jenkins has posted comments on this change. Change subject: [WIP] Change Query Plan to Use Component ID .. Patch Set 21: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1397/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2036 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5bb7c8c836dffcda48861c5fd3fe25dcb7edc659 Gerrit-PatchSet: 21 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4456/ -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1405/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1396/ (7/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5451/ (6/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1404/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2010/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6947/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. Patch Set 7: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1731/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2066 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Jianfeng Jia Gerrit-Reviewer: Taewoo Kim Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2103][STO] Too many disk components for Correlate...
Hello Anon. E. Moose #1000171, Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2066 to look at the new patch set (#7). Change subject: [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy .. [ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy - user model changes: no - storage format changes: no - interface changes: yes Details: Currently CorrelatedMergePolicy uses component Ids to ensure disk components of primary and secondary indexes are merged together, but without synchronization. However, this results in too many disk components for secondary InvertedIndex. The reason is that secondary index could miss some round of merges, if the merge policy finds out the corresponding secondary components are not available (either being merged or being flushed). Even though flow-control on secondary indexes can guarantee the secondary index would catch up the next time, it is still possible that the primary component is finialized, which leaves the secondary components which miss this round of merge are never merged again. This patch fixes this bug by: - Add the mechanism of depending operations to LSM IO operation. An operation finishes only after all depending operations have finished. - For correlated merge policy, the flush/merge of the primary index depends on all flushes/merges of secondary indexes. This ensures when the correlated policy schedules merge, all related components of all indexes are available to merge. Change-Id: I8c9bac74a7f4fed3a424e239acd352d074a270f3 --- M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedPrefixMergePolicy.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetInfo.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetResource.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/PrimaryIndexOperationTracker.java M asterixdb/asterix-common/src/test/java/org/apache/asterix/test/context/CorrelatedPrefixMergePolicyTest.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/ExternalBTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/ExternalBTreeWithBuddy.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTreeFlushOperation.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTreeMergeOperation.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTreeWithBuddyMergeOperation.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMHarness.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMIOOperation.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMIndexAccessor.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMIndexOperationContext.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractIoOperation.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndexOperationContext.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/ConstantMergePolicy.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/ExternalIndexHarness.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/FlushOperation.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMHarness.java M hyracks-fullstack/hyracks/hyracks-storage-am-ls
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5450/ (3/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Hello Anon. E. Moose #1000171, Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2025 to look at the new patch set (#35). Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. [ASTERIXDB-2115] Add Component Ids to LSM Indexes - user model changes: no - storage format changes: no - interface changes: yes Details: - Add LSMComponentId to all LSM components. Memory components receive their Id when they're created, and thus their Ids are mutable. Disk components have immutable Ids. For all created disk components, e.g., through merge, flush, or bulkload, the caller has to set the component Id properly. - To enable component Id-based acceleration, memory components of all indexes of a dataset should receive the same Id upon activation (creation). This is achieved by the CorrelatedComponentIdGenerator, which is shared across all indexes of a dataset. - Seperate diskComponent's activate from create, as other tree indexes. The activate method is called on an existing disk component, where we simply load existing structures (and component Ids), while the create method creates the new structures, and persist component Id into metadata. A design wiki for this patch: https://cwiki.apache.org/confluence/display/ ASTERIXDB/Component+Id-based+secondary-to-primary+index+acceleration Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestLsmBTreeResourceFactoryProvider.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedLSMComponentIdGenerator.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedLSMComponentIdGeneratorFactory.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedPrefixMergePolicy.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/DatasetResource.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/PrimaryIndexOperationTracker.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/ioopcallbacks/AbstractLSMIOOperationCallback.java M asterixdb/asterix-common/src/test/java/org/apache/asterix/test/context/CorrelatedPrefixMergePolicyTest.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/BTreeResourceFactoryProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/InvertedIndexResourceFactoryProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/RTreeResourceFactoryProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMIndexBulkLoadOperatorDescriptor.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMIndexBulkLoadOperatorNodePushable.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryIndexBulkLoadNodePushable.java M hyracks-fullstack/hyracks/hyracks-examples/hyracks-integration-tests/src/test/java/org/apache/hyracks/tests/am/common/LSMTreeOperatorTestHelper.java M hyracks-fullstack/hyracks/hyracks-examples/hyracks-integration-tests/src/test/java/org/apache/hyracks/tests/am/lsm/btree/LSMBTreeOperatorTestHelper.java M hyracks-fullstack/hyracks/hyracks-examples/hyracks-integration-tests/src/test/java/org/apache/hyracks/tests/am/lsm/rtree/LSMRTreeOperatorTestHelper.java M hyracks-fullstack/hyracks/hyracks-examples/hyracks-integration-tests/src/test/java/org/apache/hyracks/tests/am/lsm/rtree/LSMRTreeWithAntiMatterTuplesOperatorTestHelper.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexBulkLoadOperatorNodePushable.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/ExternalBTreeLocalResource.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/ExternalBTreeLocalResou
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/6946/ (7/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1404/ (1/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1403/ (9/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/2289/ (4/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/2098/ (5/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/2073/ (6/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2009/ (8/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/1730/ (10/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 35: Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1395/ (2/10) -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 35 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Michael Blow has posted comments on this change.
Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
..
Patch Set 2:
(1 comment)
https://asterix-gerrit.ics.uci.edu/#/c/2121/2/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/api/RPCInterface.java
File
hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/api/RPCInterface.java:
PS2, Line 97: } catch (InterruptedException e) {
: // try to wait again with a timeout
: wait(RESPONSE_TIMEOUT_ON_INTERRUPT);
: Thread.currentThread().interrupt();
: if (pending) {
: throw e;
: }
: }
> What are "all cases"?
sorry, meant that the timeout would apply to all waiting- both before and after
getting interrupted...
--
To view, visit https://asterix-gerrit.ics.uci.edu/2121
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Michael Blow
Gerrit-Reviewer: Till Westmann
Gerrit-Reviewer: abdullah alamoudi
Gerrit-HasComments: Yes
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Till Westmann has posted comments on this change.
Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
..
Patch Set 2:
(1 comment)
https://asterix-gerrit.ics.uci.edu/#/c/2121/2/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/api/RPCInterface.java
File
hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/api/RPCInterface.java:
PS2, Line 97: } catch (InterruptedException e) {
: // try to wait again with a timeout
: wait(RESPONSE_TIMEOUT_ON_INTERRUPT);
: Thread.currentThread().interrupt();
: if (pending) {
: throw e;
: }
: }
> I think we should only do this selectively for certain operations, and prob
What are "all cases"?
--
To view, visit https://asterix-gerrit.ics.uci.edu/2121
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Michael Blow
Gerrit-Reviewer: Till Westmann
Gerrit-Reviewer: abdullah alamoudi
Gerrit-HasComments: Yes
Change in asterixdb[master]: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
Michael Blow has posted comments on this change.
Change subject: [ASTERIXDB-1954][STO][RT] Add Index Drop Options
..
Patch Set 2:
(1 comment)
https://asterix-gerrit.ics.uci.edu/#/c/2121/2/hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/api/RPCInterface.java
File
hyracks-fullstack/hyracks/hyracks-ipc/src/main/java/org/apache/hyracks/ipc/api/RPCInterface.java:
PS2, Line 97: } catch (InterruptedException e) {
: // try to wait again with a timeout
: wait(RESPONSE_TIMEOUT_ON_INTERRUPT);
: Thread.currentThread().interrupt();
: if (pending) {
: throw e;
: }
: }
I think we should only do this selectively for certain operations, and probably
the operation should specify the timeout (which probably should apply in all
cases, not just interrupt?)
--
To view, visit https://asterix-gerrit.ics.uci.edu/2121
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Michael Blow
Gerrit-Reviewer: Till Westmann
Gerrit-Reviewer: abdullah alamoudi
Gerrit-HasComments: Yes
Change in asterixdb[master]: [NO ISSUE][STO] Fix memory leaks in storage
Michael Blow has submitted this change and it was merged. Change subject: [NO ISSUE][STO] Fix memory leaks in storage .. [NO ISSUE][STO] Fix memory leaks in storage - user model changes: no - storage format changes: no - interface changes: yes - Added javadocs to: -- IBufferCache -- IExtraPageBlockHelper - Moved IBufferCache.setPageDiskId -> ICachedPage.setDiskPageId - Renamed: -- IBufferCache.flushDirtyPage -> IBufferCache.flush -- IBufferCache.getNumPages -> IBufferCache.getPageBudget - Removed: -- IBufferCache.adviseWontNeed [not used] -- IBufferCache.tryPin [not used] details: - Previously, when adding a kv pair to the metadata of a memory component, we add a new Pair item to the ArrayList. After this change, we only update it if it exists. - VirtualBufferCache used to leak pages when reclaiming pages of a file after deletion. This has also been fixed. - New tests for VirtualBufferCache added: - Checks for memory budget after end of testDisjointPins - Concurrent Users pinning pages concurrently - Test for large pages and ensuring allocated large pages are accounted for through removal of cached free pages. Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15 Reviewed-on: https://asterix-gerrit.ics.uci.edu/2115 Tested-by: Jenkins Contrib: Jenkins Integration-Tests: Jenkins Reviewed-by: Murtadha Hubail --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/active/ActiveNotificationHandler.java M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml M asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/Actor.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/ComponentRollbackTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/LogMarkerTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/logging/CheckpointingTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/DiskIsFullTest.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java R hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/SingleThreadEventProcessor.java M hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties M hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/api/AbstractPointable.java A hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/util/DataUtils.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/freepage/AppendOnlyLinkedMetadataPageManager.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/freepage/LinkedMetaDataPageManager.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/impls/AbstractTreeIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMDiskComponent.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMMemoryComponent.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/DiskComponentMetadata.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/MemoryComponentMetadata.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/MultitenantVirtualBufferCache.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/NoMergePolicyFactory.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/inmemory/InMemoryInvertedIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-rtree/src/main/java/org/apache/hyracks/storage/am/rtree/impls/RTree.java M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage
Change in asterixdb[master]: [NO ISSUE][STO] Fix memory leaks in storage
Murtadha Hubail has posted comments on this change.
Change subject: [NO ISSUE][STO] Fix memory leaks in storage
..
Patch Set 10: Code-Review+2
(4 comments)
https://asterix-gerrit.ics.uci.edu/#/c/2115/10/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java
File
hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java:
PS10, Line 63: if (pageBudget == 0) {
: throw new IllegalArgumentException("Page Budget
Cannot be 0");
: }
Move this to the first line of the constructor.
PS10, Line 66: this.pageBudget = pageBudget;
Actually I recall the old logic had something to do with adding an extra page
to allow force operations to complete, either that or making the number of
pages odd. In any case, I don't think it is needed anymore.
PS10, Line 135: openFile
This being called on the assumption that a file would be opened shows you how
something is wrong with the API
PS10, Line 191: LOGGER.log(Level.INFO, "Reclaimed pages = " + reclaimedPages);
We can keep the logging but this specific log would appear out of nowhere
without any context to which file was deleted or anything.
--
To view, visit https://asterix-gerrit.ics.uci.edu/2115
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15
Gerrit-PatchSet: 10
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Dmitry Lychagin
Gerrit-Reviewer: Ian Maxon
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Luo Chen
Gerrit-Reviewer: Michael Blow
Gerrit-Reviewer: Murtadha Hubail
Gerrit-Reviewer: Till Westmann
Gerrit-Reviewer: Xikui Wang
Gerrit-Reviewer: abdullah alamoudi
Gerrit-HasComments: Yes
Change in asterixdb[master]: [WIP] Jackson parser for JSON format
Jenkins has posted comments on this change. Change subject: [WIP] Jackson parser for JSON format .. Patch Set 4: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4455/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/2076 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iacf9e496dbe2146f5eeeb1506b945991c300a7de Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Wail Alkowaileet Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Jackson parser for JSON format
Anon. E. Moose #1000171 has posted comments on this change. Change subject: [WIP] Jackson parser for JSON format .. Patch Set 4: Contrib+1 Analytics Compatibility Tests Successful Build 2878 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/2076 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iacf9e496dbe2146f5eeeb1506b945991c300a7de Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Wail Alkowaileet Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Jackson parser for JSON format
Jenkins has posted comments on this change. Change subject: [WIP] Jackson parser for JSON format .. Patch Set 4: Contrib+1 BAD Compatibility Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/2058/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/2076 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iacf9e496dbe2146f5eeeb1506b945991c300a7de Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Wail Alkowaileet Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [WIP] Jackson parser for JSON format
Jenkins has posted comments on this change. Change subject: [WIP] Jackson parser for JSON format .. Patch Set 4: BAD Compatibility Tests Started https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/2058/ -- To view, visit https://asterix-gerrit.ics.uci.edu/2076 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iacf9e496dbe2146f5eeeb1506b945991c300a7de Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Wail Alkowaileet Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2152][FUN] Enable utilizing multiple core in UDF ...
Jenkins has posted comments on this change. Change subject: [ASTERIXDB-2152][FUN] Enable utilizing multiple core in UDF evaluation .. Patch Set 3: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4453/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/2114 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id7eed5dac03c2f260507e16cf687162d65787bd1 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIXDB-2115] Add Component Ids to LSM Indexes
Anon. E. Moose #1000171 has posted comments on this change. Change subject: [ASTERIXDB-2115] Add Component Ids to LSM Indexes .. Patch Set 34: Contrib-1 Analytics Compatibility Tests Failed Build 2877 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/2025 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d Gerrit-PatchSet: 34 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Ildar Absalyamov Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Luo Chen Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
