Change in asterixdb[master]: Test the comparablity of fuzzyjoin.

[Wenhai Li (Code Review)]

Wenhai Li has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/3008

Change subject: Test the comparablity of fuzzyjoin.
..

Test the comparablity of fuzzyjoin.

The following commits from your working branch will be included:

commit a924f718b2537c856bfd470f622d8fb8879f1e07
Merge: f6aeca5 2a42da45
Author: Michael 
Date:   Fri Oct 26 13:36:12 2018 -0700

Merge branch 'master' into fuzzy

commit f6aeca5f21dfc91497b2f718815353149cd4ff03
Merge: e237045 80225e2
Author: Michael 
Date:   Fri Oct 26 13:35:08 2018 -0700

Merge branch 'master' into fuzzy

commit e237045dbcbd17bc6344d298027152b9da4ec982
Merge: 9a1a68b 1aeb8b6
Author: Michael 
Date:   Fri Sep 28 10:02:08 2018 -0700

Merge branch 'master' into fuzzy

commit 9a1a68b560c9abd30fc3c44a67c3f578a8ab9227
Merge: aa39e75 b71d9ff
Author: Michael 
Date:   Fri Sep 21 09:56:39 2018 -0700

Merge branch 'master' into fuzzy

commit aa39e75a89bda84a3ff63545b166a0611c299def
Merge: 5d5afc1 42f17d2
Author: Michael 
Date:   Fri Sep 7 10:13:50 2018 -0700

Merge branch 'master' into fuzzy

commit 5d5afc19067e3909a887cc86c7917b79be885467
Merge: 8fbe7cc 8bf93fa
Author: Michael 
Date:   Fri Aug 31 09:16:55 2018 -0700

Merge branch 'master' into fuzzy

commit 8fbe7cc54e570c71306f95040cd015482981011c
Merge: b11a785 90114df
Author: Michael 
Date:   Fri Aug 24 10:55:16 2018 -0700

Merge branch 'master' into fuzzy

commit b11a785df47fc4a7ea69611f54bf32d7151ae606
Merge: f252acc 13860fb
Author: Michael 
Date:   Mon Aug 13 09:48:42 2018 -0700

Merge branch 'master' into fuzzy

commit f252acc3eec2546bfc403c3fb2e2fa26f3383e20
Merge: aa57719 31d03f0
Author: Michael 
Date:   Fri Aug 10 09:38:53 2018 -0700

Merge branch 'master' into fuzzy

commit aa57719117ee94162a5aba521c8657ae136bce5b
Merge: 4d7c16c 0a2fb11
Author: Michael 
Date:   Wed Aug 8 02:45:56 2018 -0700

Merge branch 'master' into fuzzy

commit 4d7c16c702325aee6aa871bedb92aa128107a409
Merge: 7f4f7ac 5de12d0
Author: Michael 
Date:   Fri Aug 3 09:37:03 2018 -0700

Merge branch 'master' into fuzzy

commit 7f4f7ac2b4d54f2ef2492b248c1184e200bbbf9c
Merge: db30fa8 6b1b52c
Author: Michael 
Date:   Sat Jul 28 09:14:26 2018 -0700

Merge branch 'master' into fuzzy

commit db30fa8f10b007d44ea03321ccacc63d1ed95fc8
Merge: 4c3af5f 428a5ee
Author: Michael 
Date:   Fri Jul 27 10:53:56 2018 -0700

Merge branch 'master' into fuzzy

commit 4c3af5f0347b361b92bf1a8871cfc46652e81095
Merge: a919ce3 072da02
Author: Michael 
Date:   Tue Jul 24 09:12:07 2018 -0700

Merge branch 'master' into fuzzy

commit a919ce354053cbd90b3019bbe7f62f092388169d
Merge: 0f867e3 9a98768
Author: Michael 
Date:   Fri Jul 20 14:20:58 2018 -0700

Merge branch 'master' into fuzzy

commit 0f867e32b070b9b2dfa77bf0aadbe18d628c9f26
Merge: 57785ff 0f6df16
Author: Michael 
Date:   Sun Jul 15 02:51:10 2018 -0700

Merge branch 'master' into fuzzy

commit 57785ff1614ec2177fbd053f4d630b50f5a14127
Merge: 0974fd7 0d7974f
Author: Michael 
Date:   Fri Jul 13 03:26:41 2018 -0700

Merge branch 'master' into fuzzy

commit 0974fd783d4fdd928b6fae7f153a714d350b8c1a
Merge: c3f8654 aae5753
Author: Michael 
Date:   Fri Jul 6 10:15:40 2018 -0700

Merge branch 'master' into fuzzy

commit c3f86546bedcf8bb09abf23c5f8e68ed699938f0
Merge: 173dd2a 9087e0b
Author: Michael 
Date:   Fri Jun 29 09:59:21 2018 -0700

Merge branch 'master' into fuzzy

commit 173dd2aa1a855b4d1bc252926ef57b71f1b89948
Merge: bc6ab30 4ba23de
Author: Michael 
Date:   Fri Jun 22 10:06:01 2018 -0700

Merge branch 'master' into fuzzy

commit bc6ab3084c26ab169e3803e31365a5689848ff33
Merge: c8ddc9c 8d04d7d
Author: Michael 
Date:   Wed Jun 20 11:29:47 2018 -0700

Merge branch 'master' into fuzzy

commit c8ddc9c76841e3b9c5730e9c0ec1d29ee2a6d1ff
Merge: a9a6167 1bdf808
Author: Michael 
Date:   Fri Jun 15 10:07:45 2018 -0700

Merge branch 'master' into fuzzy

commit a9a61673ed1eb33c82e361ce6f784b951f98cf8d
Merge: 4ea5cb3 e7422b0
Author: Michael 
Date:   Fri Jun 8 09:56:42 2018 -0700

Merge branch 'master' into fuzzy

commit 4ea5cb340f4ce82d2991bc5a193aabab2ea8a1dc
Merge: eb1af35 13632a7
Author: Michael 
Date:   Fri Jun 1 10:18:22 2018 -0700

Merge branch 'master' into fuzzy

commit eb1af35a8a99d80de9e3b7fed03085fd481381be
Merge: d165d95 af17baa
Author: Michael 
Date:   Sat May 26 10:16:23 2018 -0700

Merge branch 'master' into fuzzy

commit d165d95715463a54ec9042351c2ae63fc2923e88
Merge: a7732c7 c7eb0c5
Author: Michael 
Date:   Fri May 25 10:13:24 2018 -0700

Merge branch 'master' into fuzzy

commit a7732c78b7d783603cdc6f2784c974bcadf91c9d
Merge: 3d1781b 1a6de0b
Author: Michael 
Date:   Fri May 18 01:01:04 2018 +0800

Merge branch 'master' into fuzzy

commit 3d1781b1b7a7148858caba896e56bd36de2bab85
Merge: 159dbcb 1b0bbad
Author: Michael 
Date:   Fri May 11 22:51:08 2018 +0800

Merge branch 'master' into fuzzy

commit 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#56).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/psort/cust4k-1.tbl
A asterixdb/asterix-app/data/psort/cust4k-2.tbl
A asterixdb/asterix-app/data/psort/cust4k-3.tbl
A asterixdb/asterix-app/data/psort/cust4k-4.tbl
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/OperatorResourcesComputer.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/PlanStagesGenerator.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#84).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin (has been temporarily 
disabled for sake of capacity checking round).
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#83).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin (has been temporarily 
disabled for sake of capacity checking round).
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#82).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin (has been temporarily 
disabled for sake of capacity checking round).
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#81).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#55).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/psort/cust4k-1.tbl
A asterixdb/asterix-app/data/psort/cust4k-2.tbl
A asterixdb/asterix-app/data/psort/cust4k-3.tbl
A asterixdb/asterix-app/data/psort/cust4k-4.tbl
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/APIFramework.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#54).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.
8. Use cache in Replicate (disabled).
9. Use cacheFiles in ExternalSort in Parallel sort (disabled).
10. Provide configurable cache memory budget in conf.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/psort/cust4k-1.tbl
A asterixdb/asterix-app/data/psort/cust4k-2.tbl
A asterixdb/asterix-app/data/psort/cust4k-3.tbl
A asterixdb/asterix-app/data/psort/cust4k-4.tbl
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/APIFramework.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration2.xml
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration3.xml
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration4.xml
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/CachedParallelSortRuntimeTest.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#79).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#33).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add the unit test for parallel histogram based on construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTest.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.3.query.aql
A 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#78).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#52).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/APIFramework.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration2.xml
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration3.xml
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration4.xml
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#51).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#76).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#75).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-dual-order.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/similarity/jaccard-similarity-join-right-ahead.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#74).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#73).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#50).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#49).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#48).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#72).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#69).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#68).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Use right side to build the heavy hash join on the prefix tokens from both 
sides.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#67).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Use similarity-jaccard-prefix to enable the pp+ join strategy.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#47).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce the hint "/*+ psort */" before "order by" to enable the user to 
use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in the two RangePartitioners.
8. Provide lightweight merging in the globalRgAggs.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#46).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#66).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#45).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString.

[Wenhai Li (Code Review)]

Hello Michael Blow, Preston Carman, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/806

to look at the new patch set (#63).

Change subject: RangeGenerator aggfunc for the numeric/asciiString.
..

RangeGenerator aggfunc for the numeric/asciiString.

The following commits from your working branch will be included:

Extract and review for the streaming histogram based parallel sort.
1. Add the streaming histogram for hyracks datatype.
2. Include the ASCII histogram.
3. Inline the rangegenerator inside the AggFuncs.
4. Add the testcases including the random, zipfan, gaussian primitive data 
types.
5. Support the null omittion.
Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_null/rg_int16_null.1.ddl.aql
A 

Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString.

[Wenhai Li (Code Review)]

Hello Michael Blow, Preston Carman, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/806

to look at the new patch set (#62).

Change subject: RangeGenerator aggfunc for the numeric/asciiString.
..

RangeGenerator aggfunc for the numeric/asciiString.

The following commits from your working branch will be included:

Extract and review for the streaming histogram based parallel sort.
1. Add the streaming histogram for hyracks datatype.
2. Include the ASCII histogram.
3. Inline the rangegenerator inside the AggFuncs.
4. Add the testcases including the random, zipfan, gaussian 
int/short/long/double/string.
5. Support the null omittion.
Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_null/rg_int16_null.1.ddl.aql
A 

Change in asterixdb[master]: Applied the multiway prefix-based fuzzyjoin.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#65).

Change subject: Applied the multiway prefix-based fuzzyjoin.
..

Applied the multiway prefix-based fuzzyjoin.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based join.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#64).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join.
..

Applied the multiway fuzzyjoin based on the prefix-based join.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#63).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
M asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#62).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#61).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule using prefix-based similarity join for RS cases.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csxauthors.adm
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblpauthors.adm
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#60).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT and nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#59).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#58).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#57).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.
- Increase the ManagixExecutionIT nc memory configuration so as to enable 
multi-way fuzzy join to pass the capacity check.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#32).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add unit tests for parallel histogram in terms of construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTest.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#31).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add unit tests for parallel histogram based on construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTest.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#44).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#30).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add the unit test for parallel histogram based on construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTest.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 

Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString datatype ...

[Wenhai Li (Code Review)]

Hello Michael Blow, Preston Carman, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/806

to look at the new patch set (#61).

Change subject: RangeGenerator aggfunc for the numeric/asciiString datatype 
based on parallel streaming histogram.
..

RangeGenerator aggfunc for the numeric/asciiString datatype based on parallel 
streaming histogram.

The following commits from your working branch will be included:

Extract and review for the streaming histogram based parallel sort.
1. Add the streaming histogram for hyracks datatype.
2. Include the ASCII histogram.
3. Inline the rangegenerator inside the AggFuncs.
4. Add the testcases including the random, zipfan, gaussian 
int/short/long/double/string.
5. Support the null omittion.
Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfian.tbl
A asterixdb/asterix-app/data/skew/zipfian1.tbl
A asterixdb/asterix-app/data/skew/zipfian2.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfianmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfiannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.3.query.aql
A 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#56).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#43).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#42).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#41).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString datatype ...

[Wenhai Li (Code Review)]

Hello Michael Blow, Preston Carman, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/806

to look at the new patch set (#56).

Change subject: RangeGenerator aggfunc for the numeric/asciiString datatype 
based on parallel streaming histogram.
..

RangeGenerator aggfunc for the numeric/asciiString datatype based on parallel 
streaming histogram.

The following commits from your working branch will be included:

Extract and review for the streaming histogram based parallel sort.
1. Add the streaming histogram for hyracks datatype.
2. Include the ASCII histogram.
3. Inline the rangegenerator inside the AggFuncs.
4. Add the testcases including the random, zipfan, gaussian 
int/short/long/double/string.
5. Support the null omittion.
Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_null/rg_int16_null.1.ddl.aql
A 

Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString datatype ...

[Wenhai Li (Code Review)]

Hello Michael Blow, Preston Carman, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/806

to look at the new patch set (#55).

Change subject: RangeGenerator aggfunc for the numeric/asciiString datatype 
based on parallel streaming histogram.
..

RangeGenerator aggfunc for the numeric/asciiString datatype based on parallel 
streaming histogram.

The following commits from your working branch will be included:

Extract and review for the streaming histogram based parallel sort.
1. Add the streaming histogram for hyracks datatype.
2. Include the ASCII histogram.
3. Inline the rangegenerator inside the AggFuncs.
4. Add the testcases including the random, zipfan, gaussian 
int/short/long/double/string.
5. Support the null omittion.
Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_null/rg_int16_null.1.ddl.aql
A 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#55).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto the prefix-based join due to 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#40).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#39).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in the two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#38).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#37).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in the two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#36).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#34).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#54).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#33).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#53).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#52).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#51).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#50).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#49).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin to prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 47:

(10 comments)

Revisit the improper variable names and comments.

https://asterix-gerrit.ics.uci.edu/#/c/1076/47/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 84: private static final int STAGE_1_OUTER_INDEX = 3;
> // For the index 2, you can leave it as a comment such as STAGE_2_...
Done


Line 155: // Check if the GET_ITEM function is on one of the supported 
similarity-check functions.
> is on one of -> is on the one of
Done


Line 162: // Skips this rule based on annotations.
> Like this, it would be nice all comments uses third-person singular form. I
Done


Line 203: // leftInputPKs in currrentPKs extract all the PKs derived 
from the left branch in the current fuzzyjoin.
> // leftInputPKs in currrentPKs keeps all PKs derived from the left branch i
Done


Line 207: // rightInputPKs in currentPKs extract all the PKs derived 
from the right branch in the current fuzzyjoin.
> // rightInputPKs in currrentPKs keeps all PKs derived from the left branch 
// rightInputPKs in currrentPKs keeps all PKs derived from the right branch in 
the current similarity function.

:)


Line 219: // Steps 1-2. Generate the prefix-based fuzzy jon template.
> Steps 1-2 ==> Steps 1 and 2
Done


Line 222: // Steps 3-4. Generate the prefix-based fuzzy join subplan.
> Steps 3 and 4
Done


Line 226: // Step 5. Bind the plan to the father op referred by the 
below opRef.
> father op -> parent operator
Done


Line 307: }
> String tokenizer = "";
Done


Line 317: }
> Why don't we just call FuzzyUtils.getSimThreshold at first?
Once the user gives an explicit threshold, we should call constVal to extract 
this threshold.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 47
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#32).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including
   a. Psort on full records.
   b. Psort on a join result.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in the two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#31).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in the two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#30).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in the two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#29).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#47).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#46).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 45:

(8 comments)

Thanks for reminding.

https://asterix-gerrit.ics.uci.edu/#/c/1076/45/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java:

Line 51: SequentialFirstRuleCheckFixpointRuleController 
seqFirstRuleGateKeeperDfs = new SequentialFirstRuleCheckFixpointRuleController(
> You need to apply the updated format file to avoid this violation:
I guessed you will ask this question, but my formatter in ideaj gave this form. 
I will try your method.


https://asterix-gerrit.ics.uci.edu/#/c/1076/45/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 84: private static final int STAGE_1_OUTER_INDEX = 3;
> Can we also define 2 as a certain name to understand the index? here?
We will not refer to 2, really?


Line 177: // Fuzzy similarity function extraction
> --> Extracts the expressions from the given function.
Done


Line 182: // left and right expressions are variables
> // left and right expressions are variables  --->
Done


Line 281: // Refer to fj-dblp-csx-hybrid.aql in the optTests for 
example.
> optTests -> optimized tests
Done


Line 334: // the shared token to the actually tuples of both side. 
left(right)InputPKs is used to extract those mappings
> actually -> actual, tuples of both side -> tuples of the both side
Done


Line 433: throw new 
CompilationException(ErrorCode.COMPILATION_TANSLATION_ERROR, e.getMessage());
> COMPILATION_TANSLATION_ERROR -> COMPILATION_TRANSLATION_ERROR
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/45/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java:

Line 258: @Override
> Does "hasRun" should be removed even after having the new rule controller?
Yep, at least this rule has been included by FuzzJoinRule, where we should go 
through this rule in all levels of the sub-optimized plan.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 45
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#28).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 45:

To taewoo.

I think it looks like what we want. Please recheck it after the integration 
test.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 45
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: No

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#45).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/word-fuzzyeq-jaccard_03.plan
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#27).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#26).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Add support for thetajoins.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1436

to look at the new patch set (#4).

Change subject: Add support for thetajoins.
..

Add support for thetajoins.

- Add WeightedMergeOperator to join two localRgAgg and merge onto a GlobalAgg.
- Add BinaryMergeOperatorDescriptor to merge binary ranges onto glboal one.
- Resue the ParallelSortRule to trigger binary join based on sort-merge.
- Add PriorityQueue to support sort-based join, i.e., bandjoin, thetajoin, 
after locally sorted both branches.

Change-Id: Ibb9a7f3188b6aa01b0784fa5184a2fd1a00c8f4f
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/WeightedMergeJoinPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/BinaryJoinRuntimeTest.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_band_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_theta_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_band_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_theta_join.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#44).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#43).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#42).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 39:

(21 comments)

All have been re-phased as required.
In IsomophicUtils, I use that validate var to avoid the Q warnings.

https://asterix-gerrit.ics.uci.edu/#/c/1076/39//COMMIT_MSG
Commit Message:

Line 9: - Enabled the fuzzyjoin rule.
> Enabled -> Enable
Done


Line 10: - Introduced eight existing rules in FuzzyJoinRuleCollections after 
applied the fuzzyjoin rule.
> Introduced -> Introduce
Done


Line 18:   - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
> Could you tell me what "link-like", "star-like" and "hybrid"? We can discus
Done


Line 19: - Add a running Cases for select-fuzzyjoin.
> What is select-fuzzyjoin?
select over a existing prefix-based fuzzyjoin.


Line 20: - Change the inverted-based fuzzyjoin onto prefix-based join due to 
the efficiency considerations.
> inverted-based -> inverted-index-based 
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/39/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 138: @Override public boolean rewritePost(Mutable 
opRef, IOptimizationContext context)
> public boolean ... should start at new line.
Done


Line 141: // current operator is join
> --> current operator should be a join.
Done


Line 147: // Find GET_ITEM function.
> --> Find GET_ITEM function in the join condition.
Done


Line 149: Mutable expRef = joinOp.getCondition();
> expRef -> exprRef
Done


Line 190: 
> Are we always sure that we always see two variables, not some expressions?
Till now, the composite conditions will alway place the fuzzy join condition in 
the first place after compiling (line 156). in this similarity join condition, 
we alway introduce single fields fuzzy join, which means 0: left fields' 
tokens, 1: right fields' tokens, 2: maybe a threshold has been introduced.


Line 202: // leftInputPKs in currrentPKs extract all the PKs derived 
from the left branch in the newest fuzzyjoin.
> Could you explain the meaning of "newest fuzzy join"?
newest->current


Line 212: IAType leftType = (IAType) 
context.getOutputTypeEnvironment(leftInputOp).getVarType(leftInputVar);
> You calculate this again here. leftType was already calculated in regardAsP
Has put it ahead of regardAsPrefixFuzzyJoin and changed the parameters of this 
function.


Line 258: // 2. Otherwise, we can apply this rule to its branches to 
trigger a prefix-based fuzzyjoin.
> For the above comments, I would like to suggest the following. Please check
Done


Line 259: private boolean regardAsPrefixFuzzyJoin(IOptimizationContext 
context, ILogicalOperator leftInputOp,
> regardAsPrefixFuzzyJoin -> isPrefixFuzzyJoin might be better?
Done


Line 265: // If PKs derived from the both branches are SAME as a 
previous fuzzyjoin, we treat this ~= as a select.
> we treat this ~= as a select ==> we treat this as a select over a fuzzy-joi
Done


Line 272: //Suppose we want to query on the same table on the different 
fields, i.e. A.a1 ~= B.b1 AND A.a2 ~= B.b2
> table -> dataset. Please apply this to the all places.
Done


Line 275: // Avoid the duplicated PK generation in 
findPrimaryKeysInSubplan, especially for multiway fuzzy join.
> Avoid --> Avoids
Done


Line 278: // Fail if primary keys could not be inferred.
> Fail --> Fails
Done


Line 285: // left-hand side and right-hand side of fuzzyjoin has the 
same type
> has the same type --> should be the same type.
Done


Line 428: } catch (CompilationException e) {
> CRITICAL SonarQube violation:
Done


Line 464: private Mutable 
getSimilarityExpression(Mutable expRef) {
> expRef -> exprRef
Done


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 39
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#40).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enable the fuzzyjoin rule.
- Introduce eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Case where another fuzzyjoin is regarded as select over an 
existing prefix-based fuzzyjoin.
- Change the inverted-index-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#39).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enabled the fuzzyjoin rule.
- Introduced eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Cases for select-fuzzyjoin.
- Change the inverted-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 38:

BTW, how about the AlgebricksException standard? And I think that formatting 
question in DefaultRuleSetFactory is necessary. Except for these, the rest 
questions have been resolved as required.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 38
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: No

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 38:

(18 comments)

Since the newly introduced rules inside FuzzyJoinRuleCollection is triggered 
before its following ABC rule collections (remember ABC F ABC?). You know, we 
configured CondPushDownAndJoinInferenceRuleCollection as noDfs to avoid the 
dead loop of the iterative RemoveUnusedVars/AggsRule after 
PushSelectIntoJoinRule and its related rules. I guess the variable has been 
substituted by FuzzyCollections before the second round of ABC, which results 
in this inconsistencies. But after we check that difference, we find they are 
isomorphic, i.e., the only difference aims at the variable number $101 -> $102, 
does it matter?

https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java:

Line 54: defaultLogicalRewrites
> This line should not be changed. Let's keep it as it is.
Refer to below, it was wrong. Most importantly, I cannot turn off the automatic 
grammatical checking of IntelliJ.


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 421: throw new AlgebricksException(e);
> I think this should be a CompilationException and it needs to have an error
Done


Line 428: } catch (AsterixException e) {
> We will not use AsterixException. So, after this patch set: https://asterix
Done


Line 429: throw new AlgebricksException(e);
> I think this should be a CompilationException and it needs to have an error
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
File 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql:

Line 20:  * Description: Multiple fuzzy join on three tables, with a star 
join condition.
> tables -> datasets
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
File 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql:

Line 21:  *  The base table DBLP will fuzzy join with CSX and 
propagate the results
> table -> dataset
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
File 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql:

Line 19: drop dataverse fj-dblp-csx if exists;
> Please put a comment here - what this test does.
Done


Line 26:   id: int32, 
> Remove this white space.
Done


Line 34:   id: int32, 
> Remove this white space.
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
File 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql:

Line 21:  *  The CSX and DBLP tables are used twice and will be 
propagated onto
> table -> dataset
Done


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/asterixdb/asterix-app/src/test/resources/runtimets/queries/fuzzyjoin/dblp-csx-4.1.1/word-jaccard.1.ddl.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/fuzzyjoin/dblp-csx-4.1.1/word-jaccard.1.ddl.aql:

Line 21:  *  We expect the join to be transformed into an 
prefix-based fuzzy join following with a < select.
> an -> a
"an 'less than' select" ?


https://asterix-gerrit.ics.uci.edu/#/c/1076/38/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismUtilities.java
File 
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismUtilities.java:

Line 81: // Iteratively get all the produced variables from the downstream 
operators of root.
> This returns an operator that produced the given PK variable. Right? Please
Done


Line 82: private static ILogicalOperator 
extractPKProduction(ILogicalOperator root, LogicalVariable pk)
> extractPKProduction -> getOpThatProducePK,
Done


Line 92: validate = true;
> Rather than using validate variable, you can just break here. Please check.
Done


Line 96: validate = true;
> Rather than using validate variable, you can just break here. Please check.
Done


Line 107: public static void mergeHomogeneousPK(ILogicalOperator op, 
List pks) 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#37).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enabled the fuzzyjoin rule.
- Introduced eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Cases for select-fuzzyjoin.
- Change the inverted-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#36).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enabled the fuzzyjoin rule.
- Introduced eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Cases for select-fuzzyjoin.
- Change the inverted-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 34:

(13 comments)

Two Taewoo.

Please refer to the newest version.

https://asterix-gerrit.ics.uci.edu/#/c/1076/34/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 166: List inputExps = 
simFuncExpr.getArguments();
> inputExp -> inputExprs
Done


Line 199: // leftInputPKs extract all the PKs derived from the 
downstream of the left branch.
> Please rephrase these statements. Also, can you put more explanations on "p
Done


Line 206: Collection currentPK = new HashSet<>();
> currentPK -> currentPKs
Done


Line 219: // Avoid the duplicated pk generation in 
findPrimaryKeysInSubplan, especially for multiway fuzzy join.
> pk -> PK
Done


Line 223: if (leftInputPKs == null || rightInputPKs == null) {
> Check the above SonarQube comment. leftInputPKs or rightInputPKs will not b
Done


Line 229: // left-hand side and right-hand side of "~=" has the same 
type
> "~=" -> fuzzy join
Done


Line 230: IAType left2 = TypeComputeUtils.getActualType(leftType);
> left2 -> actualLeftType
Done


Line 231: IAType right2 = TypeComputeUtils.getActualType(rightType);
> right2 -> actualRightType
Done


Line 304: throw new AlgebricksException("The given condition 
for the fuzzy join (" + joinOp.getJoinKind()
> Follow the new Exception Handling proposal. AlgebricksException should be r
Done


Line 311: // the shared token to the actually tuples of both side. 
left(right)InputPK is used to extract those mappings
> left(right)InputPK -> left(right)InputPKs
Done


Line 382: for (int i = 1; i < 4; i++) {
> initialize constant 1 as like static final int STAGE_1_INDEX = 1;
Done


Line 417: // Since the above substitution (line 150) generates the 
prefix-based join operators for the partial simJoin
> Rather than using a line number such as 150, please refer to the method or 
Done


Line 478: @Override public boolean rewritePre(Mutable 
opRef, IOptimizationContext context)
> This method is not required since the it just repeats the default method of
Done


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 34
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 35:

To Taewoo.

Except for that case requirement of leftouterjoin with extra select condition, 
the details have been revisited as required. Any composite select within 
leftoutjoin, including fuzzyjoin condition, can be regarded as its example, so 
We omitted it for sake of simplification of the comments.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 35
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: No

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#35).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enabled the fuzzyjoin rule.
- Introduced eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Cases for select-fuzzyjoin.
- Change the inverted-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Add support for thetajoins.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1436

to look at the new patch set (#3).

Change subject: Add support for thetajoins.
..

Add support for thetajoins.

- Add WeightedMergeOperator to join two localRgAgg and merge onto a GlobalAgg.
- Add BinaryMergeOperatorDescriptor to merge binary ranges onto glboal one.
- Resue the ParallelSortRule to trigger binary join based on sort-merge.
- Add PriorityQueue to support sort-based join, i.e., bandjoin, thetajoin, 
after locally sorted both branches.

Change-Id: Ibb9a7f3188b6aa01b0784fa5184a2fd1a00c8f4f
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/WeightedMergeJoinPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_band_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_theta_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_band_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_theta_join.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 

Change in asterixdb[master]: Add support for thetajoins.

[Wenhai Li (Code Review)]

Wenhai Li has uploaded a new patch set (#2).

Change subject: Add support for thetajoins.
..

Add support for thetajoins.

- Add WeightedMergeOperator to join two localRgAgg and merge onto a GlobalAgg.
- Reuse the ParallelSortRule to trigger binary join based on sort-merge.
- Add PriorityQueue to support sort-based join, i.e., bandjoin, thetajoin, 
after locally sorted both branches.

Change-Id: Ibb9a7f3188b6aa01b0784fa5184a2fd1a00c8f4f
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/WeightedMergeJoinPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_band_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_theta_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_band_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_theta_join.plan
M asterixdb/asterix-app/src/test/resources/runtimets/only.xml
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 

Change in asterixdb[master]: Add support for thetajoins. - Add WeightedMergeOperator to j...

[Wenhai Li (Code Review)]

Wenhai Li has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1436

Change subject: Add support for thetajoins. - Add WeightedMergeOperator to join 
two localRgAgg and merge onto a GlobalAgg. - Resue the ParallelSortRule to 
trigger binary join based on sort-merge. - Add PriorityQueue to support 
sort-based join, i.e., bandjoin, thetajoin, 
..

Add support for thetajoins.
- Add WeightedMergeOperator to join two localRgAgg and merge onto a GlobalAgg.
- Resue the ParallelSortRule to trigger binary join based on sort-merge.
- Add PriorityQueue to support sort-based join, i.e., bandjoin, thetajoin, 
after locally sorted both branches.

Change-Id: Ibb9a7f3188b6aa01b0784fa5184a2fd1a00c8f4f
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/WeightedMergeJoinPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_band_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_theta_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_band_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_theta_join.plan
M asterixdb/asterix-app/src/test/resources/runtimets/only.xml
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1076

to look at the new patch set (#34).

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..

Applied the multiway fuzzyjoin based on the prefix-based join and the 
selectFuzzyJoin testCases.

- Enabled the fuzzyjoin rule.
- Introduced eight existing rules in FuzzyJoinRuleCollections after applied the 
fuzzyjoin rule.
  - Add ExtractCommonExpressionRule to extract the common expressions in the 
star-like multiple fuzzyjoin substitutions.
  - Add InlineSubplanInputForNestedTupleSourceRule to translate the generated 
subplan from the similarity function-derived
substitution into join in case of nested schemas.
  - Add RemoveAssign/Variables/AggRules to iteratively remove unused 
assign/vars once FuzzyJoinRule is applied in each round.
- Add three new optimization Cases for multi-fuzzyjoin.
  - link-like multiway fuzzyjoins
  - star-like multiway fuzzyjoins
  - hybrid multiway fuzzyjoin with the both forms of fuzzyjoins.
- Add a running Cases for select-fuzzyjoin.
- Change the inverted-based fuzzyjoin onto prefix-based join due to the 
efficiency considerations.

Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/DefaultRuleSetFactory.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/FuzzyUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java
A asterixdb/asterix-app/data/dblp-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/dblp-small/dblp-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/csx-small-multi-id.txt
A asterixdb/asterix-app/data/pub-small/dblp-small-multi-id.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-hybrid.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-selflink.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-simple.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/fj-dblp-csx-star.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-hybrid.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-selflink.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-simple.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/fj-dblp-csx-star.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/ngram-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-complex/word-jaccard-check-multi-let.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ngram-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-edit-distance-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/olist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/ulist-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join-noeqjoin/word-jaccard-inline.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/issue741.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/leftouterjoin-probe-pidx-with-join-jaccard-check-idx_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance-check_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-edit-distance_04.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-fuzzyeq-jaccard_03.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_01.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_02.plan
M 
asterixdb/asterix-app/src/test/resources/optimizerts/results/inverted-index-join/ngram-jaccard-check_03.plan
M 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

[Wenhai Li (Code Review)]

Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 33:

(3 comments)

@Taewoo
All suggestions have been done. Please refer to the new version.

https://asterix-gerrit.ics.uci.edu/#/c/1076/33/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 296: prepareJoin = "join" + AQLPLUS;
> Let's define this as final static String constant like the following
Done


Line 299: prepareJoin = "loj" + AQLPLUS;
> Let's define this as final static String constant like the following
Done


Line 302: throw new IllegalStateException();
> Let's follow the error handling scheme here: https://cwiki.apache.org/confl
Done


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 33
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Enable the parallel sort in the optimizor.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1381

to look at the new patch set (#25).

Change subject: Enable the parallel sort in the optimizor.
..

Enable the parallel sort in the optimizor.

Based on the MaterializingForwardOperatorDescriptor, enable the parallel sort 
in full round.
1. Add LogicalOperator Foward to enable MaterializingForwardOperatorDescriptor 
in the logical optimizor.
2. Introduce hint "psort" before sort to enable the user to use parallel sort.
3. Add the IntroduceParallelSortRule to introduce ReplicateOperator, 
AggregateOperator, ForwardOperator before SortOperator.
4. Apply ParallelStableSortPOperator into SortOperator if the following 
conditions are satisfied:
   a. Single column sort.
   b. non-topK condition inlined.
5. Add EnforcePreAdHocReplicateRule to guarantee the consistency of the 
replicate with forward.
6. Add and partially enable the opt/rtTest cases, including:
   a. Psort on full records.
   b. Psort on a join results.
   c. Nested psort within a subplan below a secondary sort or psort.
7. Use binarySearch in two RangePartitioners.

Change-Id: I755d909b7fb34d97a1950195ed3a5a5c9d4afdd5
---
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ParallelStableSortPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceParallelSortRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
M asterixdb/asterix-algebra/src/main/javacc/AQLPlus.jj
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M asterixdb/asterix-app/pom.xml
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M asterixdb/asterix-app/src/test/resources/logging.properties
M asterixdb/asterix-app/src/test/resources/optimizerts/ignore.txt
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_group.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_join_nested.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/dynamic-connector/psort_prime.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_join_nested.plan
A 
asterixdb/asterix-app/src/test/resources/optimizerts/results/dynamic-connector/psort_prime.plan
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#29).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add the unit test for parallel histogram based on construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTests.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M asterixdb/asterix-app/pom.xml
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#28).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add the unit test for parallel histogram based on construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTest.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M asterixdb/asterix-app/pom.xml
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A 

Change in asterixdb[master]: Add the forward for ParallelSort.

[Wenhai Li (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1238

to look at the new patch set (#27).

Change subject: Add the forward for ParallelSort.
..

Add the forward for ParallelSort.

Based on the aggFuncs, enable the parallel sort by:
1. Maintaining the rangeMap within the HyracksJobletConext
2. Setup forward operator to connect the partitioning and sorting operators.
3. Add the unit test for parallel histogram based on construction-merge.
4. Add the delimited parser to inline the tags onto file scan.
5. Add four partitioners for the schema-driven assigner/rgAgg in asterixTests.

Change-Id: Ifc3ca935f569d33bda81e4c3bcf7daad8749d84c
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushAggFuncIntoStandaloneAggregateRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/FunctionCollection.java
A asterixdb/asterix-app/data/skew/zipfan1.tbl
A asterixdb/asterix-app/data/skew/zipfan2.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_1.tbl
A asterixdb/asterix-app/data/skew/zipfanmissing_2.tbl
A asterixdb/asterix-app/data/skew/zipfannull_1.tbl
A asterixdb/asterix-app/data/skew/zipfannull_2.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part1.tbl
A asterixdb/asterix-app/data/tpch0.001/lineitem-part2.tbl
M asterixdb/asterix-app/pom.xml
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/AsterixAlgebricksHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/runtime/ParallelSortRuntimeTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/TestPartitionComputerFactory.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_float/avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int16/avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int32/avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int64/avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/avg_int8/avg_int8.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/count_01/count_01.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-avg_01/global-avg_01.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_double/local-avg_double.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_float/local-avg_float.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int16/local-avg_int16.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int32/local-avg_int32.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int64/local-avg_int64.3.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-avg_int8/local-avg_int8.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double/rg_double.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_double_missing/rg_double_missing.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16/rg_int16.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg_int16_missing/rg_int16_missing.2.update.aql
A