[jira] [Commented] (ASTERIXDB-1556) Hash Table used by External hash group-by doesn't conform to the budget.

2016-11-07 Thread Michael J. Carey (JIRA)

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646596#comment-15646596
 ] 

Michael J. Carey commented on ASTERIXDB-1556:
-

Can we pre-reserve the space before doing the actual ops?





> Hash Table used by External hash group-by doesn't conform to the budget.
> 
>
> Key: ASTERIXDB-1556
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Taewoo Kim
>Assignee: Taewoo Kim
>Priority: Critical
>  Labels: soon
> Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, 
> 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf
>
>
> When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > 
> 2), the system generates an out-of-memory exception. 
> Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is 
> translated into massive number of operators (more than 200 operators in the 
> plan for a 3-way fuzzy join), it could generate out-of-memory exception.
> /// Update: as the discussion goes, we found that hash table in the external 
> hash group by doesn't conform to the frame limit. So, an out of memory 
> exception happens during the execution of an external hash group by operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Change in asterixdb[master]: ASTERIXDB-1727: Fix the issue with multiple aggregates in on...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1727: Fix the issue with multiple aggregates in one 
group-by
..


Patch Set 2: Integration-Tests-1

Integration Tests Timed Out

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1105/ 
: ABORTED

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1343
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb5619cc3ece00ab83962d53e004f203684df9ee
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1727: Fix the issue with multiple aggregates in on...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1727: Fix the issue with multiple aggregates in one 
group-by
..


Patch Set 2:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1105/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1343
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb5619cc3ece00ab83962d53e004f203684df9ee
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1727: Fix the issue with multiple aggregates in on...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1727: Fix the issue with multiple aggregates in one 
group-by
..


Patch Set 2:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1343
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb5619cc3ece00ab83962d53e004f203684df9ee
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1727: Fix the issue with multiple aggregates in on...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1727: Fix the issue with multiple aggregates in one 
group-by
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3267/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1343
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb5619cc3ece00ab83962d53e004f203684df9ee
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1727: Fix the issue with multiple aggregates in on...

2016-11-07 Thread Taewoo Kim (Code Review)
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1343

to look at the new patch set (#2).

Change subject: ASTERIXDB-1727: Fix the issue with multiple aggregates in one 
group-by
..

ASTERIXDB-1727: Fix the issue with multiple aggregates in one group-by

 - Although the hash group-by hint is given, if multiple aggregate operators
   exist in the subplan of group-by, the physical operator of the given group-by
   should not be set as EXTERNAL_GROUP_BY since we don't support multiple 
aggregates
   in the external group by physical operator.

Change-Id: Ifb5619cc3ece00ab83962d53e004f203684df9ee
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/query-ASTERIXDB-1727/query-ASTERIXDB-1727.1.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate/query-ASTERIXDB-1727/query-ASTERIXDB-1727.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java
5 files changed, 94 insertions(+), 1 deletion(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/43/1343/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1343
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifb5619cc3ece00ab83962d53e004f203684df9ee
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Yingyi Bu 


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 3: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1104/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Address Don's comments in the expression doc.

2016-11-07 Thread Yingyi Bu (Code Review)
Yingyi Bu has posted comments on this change.

Change subject: Address Don's comments in the expression doc.
..


Patch Set 1:

(1 comment)

https://asterix-gerrit.ics.uci.edu/#/c/1327/1/asterixdb/asterix-doc/src/main/markdown/sqlpp/2_expr.md
File asterixdb/asterix-doc/src/main/markdown/sqlpp/2_expr.md:

Line 196: A type error will be raised if the first expression in a quantified 
expression does not return a collection.
> How is this done in other languages? I think that XQuery/JSONiq would not c
AQL does the same as SQL++.
N1QL does implicit scalar to singleton coercion.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1327
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I224a706aa987a0d938ab22b9ae28660ef6433991
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Yingyi Bu 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: Yes


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 3:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1104/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 3:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 3:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3266/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Akshay Manchale (Code Review)
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1334

to look at the new patch set (#3).

Change subject: Fixed issues with Replication tests
..

Fixed issues with Replication tests

- Changed the way the log reader writes the newValue
- Fixed bug with creating recovery plan
- Fixed offset calculation

Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
M 
asterixdb/asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
M 
asterixdb/asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
M 
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/util/ClusterStateManager.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java
6 files changed, 280 insertions(+), 13 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/34/1334/3
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 


[jira] [Commented] (ASTERIXDB-1556) Hash Table used by External hash group-by doesn't conform to the budget.

2016-11-07 Thread Taewoo Kim (JIRA)

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645756#comment-15645756
 ] 

Taewoo Kim commented on ASTERIXDB-1556:
---

I'm now working on this idea: Let the D and H share the buffer pool (the same 
to [~che...@gmail.com]'s idea).

It turns out that allocating frames from the same buffer pool is a good thing 
that we don't need to check H(hash table byte size) + D(data table byte size) > 
M(memory budget). But, it now generates a different issue - atomicity. 
Currently, we don't have this issue because data table and hash table get the 
memory frames from the different source. That is, they never fail. However, 
when we insert a tuple from an incoming frame, we 1) insert a tuple to data 
table and get the tuple pointer(frame index, offset). Then 2) insert this tuple 
pointer to the hash table. At this time, there might not be enough memory so 
that the operation 1) succeeds and 2) might fail. In this case, we need to 
revert the effect of 1). Also, 2) consists of multiple steps in case if a hash 
slot needs to be migrated due to slot overflow, partial steps done in 2) need 
to be reversed, too. 

> Hash Table used by External hash group-by doesn't conform to the budget.
> 
>
> Key: ASTERIXDB-1556
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Taewoo Kim
>Assignee: Taewoo Kim
>Priority: Critical
>  Labels: soon
> Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, 
> 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf
>
>
> When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > 
> 2), the system generates an out-of-memory exception. 
> Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is 
> translated into massive number of operators (more than 200 operators in the 
> plan for a 3-way fuzzy join), it could generate out-of-memory exception.
> /// Update: as the discussion goes, we found that hash table in the external 
> hash group by doesn't conform to the frame limit. So, an out of memory 
> exception happens during the execution of an external hash group by operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 2: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1103/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Tests for ASTERIXDB-1725

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Tests for ASTERIXDB-1725
..


Patch Set 2: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1102/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1344
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8b241acebbdcd658e734a43d03259a1ea24d8b57
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Ian Maxon 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 2:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1103/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 2:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Fixed issues with Replication tests
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3265/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No


Change in asterixdb[master]: Fixed issues with Replication tests

2016-11-07 Thread Akshay Manchale (Code Review)
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1334

to look at the new patch set (#2).

Change subject: Fixed issues with Replication tests
..

Fixed issues with Replication tests

- Fixed offset calculation

Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java
M 
asterixdb/asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm
M 
asterixdb/asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm
M 
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/util/ClusterStateManager.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java
6 files changed, 279 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/34/1334/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1334
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Akshay Manchale 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 


Change in asterixdb[master]: Tests for ASTERIXDB-1725

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Tests for ASTERIXDB-1725
..


Patch Set 2:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1102/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1344
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8b241acebbdcd658e734a43d03259a1ea24d8b57
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Ian Maxon 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-HasComments: No


Change in asterixdb[master]: Tests for ASTERIXDB-1725

2016-11-07 Thread Ian Maxon (Code Review)
Ian Maxon has posted comments on this change.

Change subject: Tests for ASTERIXDB-1725
..


Patch Set 1:

(1 comment)

https://asterix-gerrit.ics.uci.edu/#/c/1344/1/asterixdb/asterix-installer/src/test/java/org/apache/asterix/installer/test/AsterixRestartIT.java
File 
asterixdb/asterix-installer/src/test/java/org/apache/asterix/installer/test/AsterixRestartIT.java:

Line 83: for (TestCaseContext testCaseCtx : testCaseCollection) {
> Can we change this to be a parameterized test, so that these aren't collaps
Sure, done in next patch (I think?).


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1344
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8b241acebbdcd658e734a43d03259a1ea24d8b57
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Ian Maxon 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-HasComments: Yes


Change in asterixdb[master]: ASTERIXDB-1714: Eliminate dependency on org.json

2016-11-07 Thread Ian Maxon (Code Review)
Ian Maxon has posted comments on this change.

Change subject: ASTERIXDB-1714: Eliminate dependency on org.json
..


Patch Set 5: Code-Review+2

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1318
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I3ada73e15c781a77b5b3019f246e1a1fc23a2377
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Till Westmann 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No


Change in asterixdb[master]: Tests for ASTERIXDB-1725

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: Tests for ASTERIXDB-1725
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3264/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1344
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8b241acebbdcd658e734a43d03259a1ea24d8b57
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-HasComments: No


Change in asterixdb[master]: Tests for ASTERIXDB-1725

2016-11-07 Thread Ian Maxon (Code Review)
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1344

to look at the new patch set (#2).

Change subject: Tests for ASTERIXDB-1725
..

Tests for ASTERIXDB-1725

Just inserts a really big record, stops, and starts again.
This should always work, but for the bug, it'd cause a nullpointer
upon querying the dataset.

Change-Id: I8b241acebbdcd658e734a43d03259a1ea24d8b57
---
M 
asterixdb/asterix-installer/src/test/java/org/apache/asterix/installer/test/AsterixRestartIT.java
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/hugeobject.adm
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/queries/storage-restart/issue-1725/issue-1725.1.ddl.aql
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/queries/storage-restart/issue-1725/issue-1725.2.update.aql
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/queries/storage-restart/issue-1725/issue-1725.3.mgx.aql
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/queries/storage-restart/issue-1725/issue-1725.4.mgx.aql
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/queries/storage-restart/issue-1725/issue-1725.5.query.aql
A 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/results/storage-restart/issue-1725/issue-1725.5.aql
M 
asterixdb/asterix-installer/src/test/resources/integrationts/restart/testsuite.xml
9 files changed, 141 insertions(+), 23 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/44/1344/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1344
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8b241acebbdcd658e734a43d03259a1ea24d8b57
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 


Change in asterixdb[master]: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure ...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure 
function calls
..


Patch Set 22: Integration-Tests-1

Integration Tests Failed

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1101/ 
: UNSTABLE

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1057
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2dec322b30835625430c06acd7626d902bada137
Gerrit-PatchSet: 22
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Preston Carman 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


[jira] [Resolved] (ASTERIXDB-1732) Sporadic deadlock in LogManager

2016-11-07 Thread Till (JIRA)

 [ 
https://issues.apache.org/jira/browse/ASTERIXDB-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till resolved ASTERIXDB-1732.
-
Resolution: Fixed

https://github.com/apache/asterixdb/commit/d292151eb4f0983705e60dee5d43cea8d33d486e

> Sporadic deadlock in LogManager
> ---
>
> Key: ASTERIXDB-1732
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1732
> Project: Apache AsterixDB
>  Issue Type: Bug
>  Components: AsterixDB, Transactions
>Reporter: Till
>Assignee: Abdullah Alamoudi
>Priority: Critical
>  Labels: sporadic-error
>
> The LogBuffer has a race condition if the log buffer capacity is exceeded. If 
> this happens, we can lose log pages for large records. In that case the log 
> is corrupted and it leads to a deadlock when the LogManager switches to a new 
> log file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (ASTERIXDB-1732) Sporadic deadlock in LogManager

2016-11-07 Thread Till (JIRA)

 [ 
https://issues.apache.org/jira/browse/ASTERIXDB-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till updated ASTERIXDB-1732:

Labels: sporadic-error  (was: )

> Sporadic deadlock in LogManager
> ---
>
> Key: ASTERIXDB-1732
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1732
> Project: Apache AsterixDB
>  Issue Type: Bug
>  Components: AsterixDB, Transactions
>Reporter: Till
>Assignee: Abdullah Alamoudi
>Priority: Critical
>  Labels: sporadic-error
>
> The LogBuffer has a race condition if the log buffer capacity is exceeded. If 
> this happens, we can lose log pages for large records. In that case the log 
> is corrupted and it leads to a deadlock when the LogManager switches to a new 
> log file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (ASTERIXDB-1732) Sporadic deadlock in LogManager

2016-11-07 Thread Till (JIRA)

 [ 
https://issues.apache.org/jira/browse/ASTERIXDB-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till updated ASTERIXDB-1732:

Summary: Sporadic deadlock in LogManager  (was: Deadlock in LogManager)

> Sporadic deadlock in LogManager
> ---
>
> Key: ASTERIXDB-1732
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1732
> Project: Apache AsterixDB
>  Issue Type: Bug
>  Components: AsterixDB, Transactions
>Reporter: Till
>Assignee: Abdullah Alamoudi
>Priority: Critical
>  Labels: sporadic-error
>
> The LogBuffer has a race condition if the log buffer capacity is exceeded. If 
> this happens, we can lose log pages for large records. In that case the log 
> is corrupted and it leads to a deadlock when the LogManager switches to a new 
> log file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (ASTERIXDB-1732) Deadlock in LogManager

2016-11-07 Thread Till (JIRA)
Till created ASTERIXDB-1732:
---

 Summary: Deadlock in LogManager
 Key: ASTERIXDB-1732
 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1732
 Project: Apache AsterixDB
  Issue Type: Bug
  Components: AsterixDB, Transactions
Reporter: Till
Assignee: Abdullah Alamoudi
Priority: Critical


The LogBuffer has a race condition if the log buffer capacity is exceeded. If 
this happens, we can lose log pages for large records. In that case the log is 
corrupted and it leads to a deadlock when the LogManager switches to a new log 
file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (ASTERIXDB-1725) Root page location calculation does not account for multipages

2016-11-07 Thread Ian Maxon (JIRA)

 [ 
https://issues.apache.org/jira/browse/ASTERIXDB-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ian Maxon resolved ASTERIXDB-1725.
--
Resolution: Fixed

Tests for this are incoming but it's fixed.

> Root page location calculation does not account for multipages
> --
>
> Key: ASTERIXDB-1725
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1725
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Ian Maxon
>Assignee: Ian Maxon
>
> The root page calculation is based on a fixed number of pages. This is 
> usually fine, unless the root page happens to be a leaf page. In this case it 
> may consist of any number of pages due to large object storage. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (ASTERIXDB-1731) stopcc should stop all ncdrivers

2016-11-07 Thread Till (JIRA)

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644910#comment-15644910
 ] 

Till commented on ASTERIXDB-1731:
-

Seems that is would be safe to trigger the same shutdown procedure that the 
HTTP API triggers on a regular shutdown of the CC.

> stopcc should stop all ncdrivers 
> -
>
> Key: ASTERIXDB-1731
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1731
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Yingyi Bu
>Assignee: Michael Blow
>
> Currently, by ncservice, startcc automatically start ncdrivers. However, 
> stopcc doesn't automatically stop all ncdrivers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (ASTERIXDB-1731) stopcc should stop all ncdrivers

2016-11-07 Thread Yingyi Bu (JIRA)
Yingyi Bu created ASTERIXDB-1731:


 Summary: stopcc should stop all ncdrivers 
 Key: ASTERIXDB-1731
 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1731
 Project: Apache AsterixDB
  Issue Type: Bug
Reporter: Yingyi Bu
Assignee: Michael Blow


Currently, by ncservice, startcc automatically start ncdrivers. However, stopcc 
doesn't automatically stop all ncdrivers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (ASTERIXDB-1570) Sporadic test failure "Unable to find free page in buffer cache after 1000 cycles"

2016-11-07 Thread ASF subversion and git services (JIRA)

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644899#comment-15644899
 ] 

ASF subversion and git services commented on ASTERIXDB-1570:


Commit 9af8a4b559cb6274d1842d786725705f404155d2 in asterixdb's branch 
refs/heads/master from [~mblow]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=9af8a4b ]

ASTERIXDB-1570: "Unable to find free page in buffer cache after 1000 cycles"

Increase test buffer cache size to reliably accomodate 20M values

Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1346
Sonar-Qube: Jenkins 
Tested-by: Jenkins 
Integration-Tests: Jenkins 
Reviewed-by: Till Westmann 


> Sporadic test failure "Unable to find free page in buffer cache after 1000 
> cycles"
> --
>
> Key: ASTERIXDB-1570
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1570
> Project: Apache AsterixDB
>  Issue Type: Bug
>  Components: Storage
>Reporter: Yingyi Bu
>Assignee: Michael Blow
>Priority: Critical
>  Labels: soon
>
> It happened on my recent jenkins job:
> {noformat}
> Error Message
> Test 
> "src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.3.query.aql"
>  FAILED!
> Stacktrace
> java.lang.Exception: Test 
> "src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.3.query.aql"
>  FAILED!
>   at 
> org.apache.asterix.test.aql.TestExecutor.executeHttpRequest(TestExecutor.java:278)
>   at 
> org.apache.asterix.test.aql.TestExecutor.executeQuery(TestExecutor.java:288)
>   at 
> org.apache.asterix.test.aql.TestExecutor.executeTest(TestExecutor.java:538)
>   at 
> org.apache.asterix.test.aql.TestExecutor.executeTest(TestExecutor.java:785)
>   at 
> org.apache.asterix.test.runtime.ExecutionTest.test(ExecutionTest.java:110)
> Standard Error
> org.apache.hyracks.api.exceptions.HyracksDataException: Unable to find free 
> page in buffer cache after 1000 cycles (buffer cache undersized?)
>   at 
> org.apache.hyracks.storage.common.buffercache.BufferCache.getPageLoop(BufferCache.java:1287)
>   at 
> org.apache.hyracks.storage.common.buffercache.BufferCache.confiscatePage(BufferCache.java:1154)
>   at 
> org.apache.hyracks.storage.common.buffercache.BufferCache.confiscatePage(BufferCache.java:1141)
>   at 
> org.apache.hyracks.storage.am.bloomfilter.impls.BloomFilter$BloomFilterBuilder.allocateAndInitMetaDataPage(BloomFilter.java:253)
>   at 
> org.apache.hyracks.storage.am.bloomfilter.impls.BloomFilter$BloomFilterBuilder.end(BloomFilter.java:283)
>   at 
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.flush(LSMBTree.java:481)
>   at 
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.flush(LSMHarness.java:434)
>   at 
> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.flush(LSMTreeIndexAccessor.java:105)
>   at 
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeFlushOperation.call(LSMBTreeFlushOperation.java:73)
>   at 
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeFlushOperation.call(LSMBTreeFlushOperation.java:34)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> org.apache.hyracks.api.exceptions.HyracksDataException: 
> org.apache.hyracks.api.exceptions.HyracksDataException: Unable to find free 
> page in buffer cache after 1000 cycles (buffer cache undersized?)
>   at 
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
>   at 
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
>   at org.apache.hyracks.control.nc.Task.run(Task.java:263)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: 
> org.apache.hyracks.api.exceptions.HyracksDataException: Unable to find free 
> page in buffer cache after 1000 cycles (buffer cache undersized?)
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at 

Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Michael Blow (Code Review)
Michael Blow has submitted this change and it was merged.

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..


ASTERIXDB-1570: "Unable to find free page in buffer cache after 1000 cycles"

Increase test buffer cache size to reliably accomodate 20M values

Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1346
Sonar-Qube: Jenkins 
Tested-by: Jenkins 
Integration-Tests: Jenkins 
Reviewed-by: Till Westmann 
---
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
M 
asterixdb/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M 
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
4 files changed, 12 insertions(+), 12 deletions(-)

Approvals:
  Till Westmann: Looks good to me, approved
  Jenkins: Verified; No violations found; Verified



diff --git 
a/asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml 
b/asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
index ef0cdf2..90dbe5f 100644
--- a/asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
+++ b/asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
@@ -74,15 +74,15 @@
 storage.buffercache.pagesize
 32KB
 The page size in bytes for pages in the buffer cache.
-  (Default = "32768" // 32KB)
+  (Default = "128KB")
 
   
   
 storage.buffercache.size
-32MB
+48MB
 The size of memory allocated to the disk buffer cache.
-  The value should be a multiple of the buffer cache page size(Default
-  = "33554432" // 32MB)
+  The value should be a multiple of the buffer cache page size
+  (Default = "512MB")
 
   
   
diff --git 
a/asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
 
b/asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
index 649f1e8..a20b1a7 100644
--- 
a/asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
+++ 
b/asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
@@ -55,7 +55,7 @@
 "plot.activate": false,
 "storage.buffercache.maxopenfiles": 2147483647,
 "storage.buffercache.pagesize": 32768,
-"storage.buffercache.size": 33554432,
+"storage.buffercache.size": 50331648,
 "storage.lsm.bloomfilter.falsepositiverate": 0.01,
 "storage.memorycomponent.globalbudget": 536870912,
 "storage.memorycomponent.numcomponents": 2,
diff --git 
a/asterixdb/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
 
b/asterixdb/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
index c6da1c0..4080325 100644
--- 
a/asterixdb/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
+++ 
b/asterixdb/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
@@ -185,8 +185,7 @@
 }
 lineActual = readerActual.readLine();
 if (lineActual != null) {
-throw new ComparisonException(
-"Result for " + scriptFile + " changed at line " + num 
+ ":\n< \n> " + lineActual);
+throwLineChanged(scriptFile, "", lineActual, num);
 }
 } catch (Exception e) {
 System.err.println("Actual results file: " + 
actualFile.toString());
diff --git 
a/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
 
b/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
index ed33cc4..8ff9ab6 100644
--- 
a/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
+++ 
b/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
@@ -1248,7 +1248,7 @@
 
 private ICachedPage getPageLoop(long dpid, int multiplier, boolean 
confiscate)
 throws HyracksDataException {
-final long startingPinCount = masterPinCount.get();
+final long startingPinCount = DEBUG ? masterPinCount.get() : -1;
 int cycleCount = 0;
 try {
 while (true) {
@@ -1291,15 +1291,16 @@
 if (cycleCount > MAX_PIN_ATTEMPT_CYCLES) {
 cycleCount = 0; // suppress warning below
   

Change in asterixdb[master]: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure ...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure 
function calls
..


Patch Set 22:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1101/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1057
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2dec322b30835625430c06acd7626d902bada137
Gerrit-PatchSet: 22
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Preston Carman 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure ...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure 
function calls
..


Patch Set 22:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1057
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2dec322b30835625430c06acd7626d902bada137
Gerrit-PatchSet: 22
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Preston Carman 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure ...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1608, ASTERIXDB-1617 Match user query for nonpure 
function calls
..


Patch Set 22:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3263/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1057
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2dec322b30835625430c06acd7626d902bada137
Gerrit-PatchSet: 22
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Preston Carman 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Till Westmann (Code Review)
Till Westmann has posted comments on this change.

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..


Patch Set 2: Code-Review+2

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1346
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Michael Blow 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-HasComments: No


Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString datatype ...

2016-11-07 Thread Wenhai Li (Code Review)
Wenhai Li has posted comments on this change.

Change subject: RangeGenerator aggfunc for the numeric/asciiString datatype 
based on parallel streaming histogram.
..


Patch Set 50:

Mike B. +1 sorry!

Hi, Yingyi and Preston.
I didn't know you cann't see the comments without publishing. :( Hope it's not 
too late.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/806
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01
Gerrit-PatchSet: 50
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Jianfeng Jia 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Preston Carman 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Wenhai Li 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No


Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString datatype ...

2016-11-07 Thread Wenhai Li (Code Review)
Wenhai Li has posted comments on this change.

Change subject: RangeGenerator aggfunc for the numeric/asciiString datatype 
based on parallel streaming histogram.
..


Patch Set 21:

(21 comments)

Hi, Yingyi and Preston.
I didn't know you cann't see the comments without publishing. :( Hope it's not 
too late.

https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.ddl.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.1.ddl.aql:

Line 20: 
> remove this file
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.2.update.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.2.update.aql:

Line 17:  * under the License.
> remove this file
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.3.query.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/global-rg/global-rg.3.query.aql:

Line 19: use dataverse test;
> rename this file to global-rg.1.query.aql
Done


Line 22:   for $x in [1.0, 2.0, double("3.0"), 3.1, 3.2, 3.3, 3.4] 
> WS
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.ddl.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.1.ddl.aql:

Line 20: 
> remove this file.
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.2.update.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.2.update.aql:

Line 16:  * specific language governing permissions and limitations
> remove this file
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.3.query.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/local-rg/local-rg.3.query.aql:

Line 13:  * software distributed under the License is distributed on an
> rename this file to local-rg.1.query.aql
Done


Line 22:   for $x in [1.0, 2.0, double("3.0"), 3.1, 3.2, 3.3, 3.4] 
> WS
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.ddl.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.1.ddl.aql:

Line 8:  * with the License.  You may obtain a copy of the License at
> remove this file
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.2.update.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.2.update.aql:

Line 17:  * under the License.
> remove this file.
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.3.query.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double-null/rg-double-null.3.query.aql:

Line 19: use dataverse test;
> rename this file to rg-double-null.1.query.aql
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.ddl.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.1.ddl.aql:

Line 21: create dataverse test;
> remove this file
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.2.update.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.2.update.aql:

Line 12:  * Unless required by applicable law or agreed to in writing,
> remove this file
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.3.query.aql
File 
asterixdb/asterix-app/src/test/resources/runtimets/queries/aggregate/rg-double/rg-double.3.query.aql:

Line 20: set partitions '2'
> rename this file to rg-double.3.query.aql.
Done


Line 22:   for $x in [1.0, 2.0, double("3.0"), 3.1, 3.2, 3.3, 3.4] 
> WS
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/21/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/AsterixBuiltinFunctions.java
File 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/AsterixBuiltinFunctions.java:

Line 225: "ceiling", 1);
> code style doesn't seem right.
Done



Change in asterixdb[master]: RangeGenerator aggfunc for the numeric/asciiString datatype ...

2016-11-07 Thread Wenhai Li (Code Review)
Wenhai Li has posted comments on this change.

Change subject: RangeGenerator aggfunc for the numeric/asciiString datatype 
based on parallel streaming histogram.
..


Patch Set 36:

(13 comments)

Hi, Yingyi and Preston.
I didn't know you cann't see the comments without publishing. :( Hope it's not 
too late.

https://asterix-gerrit.ics.uci.edu/#/c/806/36/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/typecomputer/impl/GlobalRangeGeneratorTypeComputer.java
File 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/typecomputer/impl/GlobalRangeGeneratorTypeComputer.java:

Line 30: public class GlobalRangeGeneratorTypeComputer  extends 
AbstractResultTypeComputer {
> extra space before extends
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarRangeGeneratorAggregateDescriptor.java
File 
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarRangeGeneratorAggregateDescriptor.java:

Line 28: public final static FunctionIdentifier FID = 
AsterixBuiltinFunctions.SCALAR_RG;
> final static -> static final
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/std/AbstractRangeGeneratorAggregateFunction.java
File 
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/std/AbstractRangeGeneratorAggregateFunction.java:

Line 323: for (int i = 0; i < listItems.size(); i++) {
> Seems it might be a little cleaner to use the enhanced for loop:
Done


Line 384: for (int i = 0; i < rangeMap.size(); i++) {
> Seems it might be a little cleaner to use the enhanced for loop here
Done


Line 409: for (int i = 0; i < rangeMap.size(); i++) {
> Seems it might be a little cleaner to use the enhanced for loop here
Done


Line 442: translateVal = (ShortPointable) 
BytePointable.FACTORY.createPointable();
> bad cast (BytePointable)
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/std/LocalRangeGeneratorAggregateFunction.java
File 
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/std/LocalRangeGeneratorAggregateFunction.java:

Line 42: finishPartialResults(result);
> making sure this is the right method to call (partial)
This is right since the the total merging part will be execute within the 
GlobalRangeGeneratorAggregateFunction.


https://asterix-gerrit.ics.uci.edu/#/c/806/36/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/IHeapList.java
File 
hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/IHeapList.java:

Line 24: public void increase(int index, K key) throws HyracksDataException;
> can remove redundant 'public' modifier (all methods)
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/IHistogram.java
File 
hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/IHistogram.java:

Line 27:  * @author michael
> remove @author
Done


Line 30: public enum HistogramType {
> can remove redundant 'public' modifier (all enums & methods)
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/IQuantileList.java
File 
hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/IQuantileList.java:

Line 27: public int size();
> can remove redundant 'public' modifier (all methods)
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/structures/DominantQuantile.java
File 
hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/structures/DominantQuantile.java:

Line 70: return super.hashCode();
> Hashcode function should be implemented or this class may not behave as exp
Done


https://asterix-gerrit.ics.uci.edu/#/c/806/36/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/structures/GenericQuantile.java
File 
hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/range/structures/GenericQuantile.java:

Line 73: return super.hashCode();
> Hashcode function should be implemented or this class may not behave as exp
Done


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/806
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I450d0962fbeacfb2b6ab9fae0750f025ef17ba01

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

2016-11-07 Thread Wenhai Li (Code Review)
Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 21:

(10 comments)

@Taewoo,

Sorry, I did not know you cann't see the comments without publishing. :)

OK, published. Maybe we can talk with some detail with running example. You 
know, for those specified detail, these pessimistic methods can always forbid 
the exception of the dynamic variables generation.

https://asterix-gerrit.ics.uci.edu/#/c/1076/21/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 197: // To handle multiple fuzzyjoin conditions on the same table 
pair, this rule differentiate the PKs
> What do we mean by [the same table pair] here?
Query example:
use dataverse fuzzytest;
for $d in dataset DBLP
for $t in dataset CSX
where word-tokens($d.title) ~= word-tokens($t.title) and 
word-tokens($d.authors) ~= word-tokens($t.authors)
return {"did": $d.tid, "tid": $t.tid}

Explain in general:

The first round has the following functional dependencies:
1. $d.title -> $d.tid
2. $t.title -> $t.tid
which means $d.title is derived from table $d and $t.title is derived from $t, 
respectively.

After iteration, in the second round, we have the following functional 
dependencies:
1. $d.authors -> $d.tid
2. $t.authors -> $t.tid
the both right parts have been maintained in the previousPK.

Result:
In this context, we just give two fuzzy join condition on a same table pair, 
and the second fuzzy join SHOULD be explained as a fuzzy select based on the 
result of the first fuzzy join.

Handle strategy:
Just omit the second fuzzy join, other than explain it as another fuzzy join to 
avoid the wrong substitution based on the fixed template. (Since we have 
substituted the two table branches in the first fuzzy join.)


Line 207: Set currentPK = new HashSet<>();
> I'm confused about currentPK and previousPK concept. Can you explain more?
In general, each round of potential substitution will scan all its branch 
variables to look forward where are they coming from.

currentPK is the primary key of all the primary keys of the current ~='s 
branches.

previousPK is the primary key of all the primary keys of the 
scanned/substituted ~='s branches.

If they are equal, we claim it's the duplicated fuzzyjoin conditions on a same 
table pair.

i.e.

use dataverse fuzzytest; for $d in dataset DBLP for $t in dataset CSX where 
word-tokens($d.title) ~= word-tokens($t.title) and word-tokens($d.authors) ~= 
word-tokens($t.authors) return {"did": $d.tid, "tid": $t.tid}

$d.title and $t.title as well as their PKs are the previous derivations, and 
$d.authors and $t.authors are the current derivations.


Line 210: // If PKs derived from the both branches are SAME as a 
previous fuzzyjoin, we treat this ~= as a select.
> Here, "previous fuzzy join" means? Can you present an example?
Reference the comment on 207's word-tokens($d.title) ~= word-tokens($t.title).


Line 251: ConstantExpression constExpr = (ConstantExpression) inputExp2;
> The reason of this change - not using FuzzyUtils.getSimThreshold()?
At least one case involved:

similarity-jaccard() <> threshold

to get threshold, I think FuzzyUtils.getSimThreshold is not enough.


Line 268: break;
> Have we fixed the bug that mentioned in the previous TODO? Can we explain m
If only permuting the three for clauses in the mentioned testCase, the results 
in this code-branch are consistent. Also, if we change the join conditions in 
this query, I think it's not an issues, but a semantic problem. I guess the old 
issues as commented left-red is derived from the flatten process or order 
issue. But anyway, it's disappeared in this branch on current master.


Line 317: translator.addVariableToMetaScope(new 
VarIdentifier("$$LEFT_0"), leftInputVar);
> What's the difference between # and $$? I think I saw this in the Vernica's
Also, this issue is derived from the new master at about one year ago. In 
short, "#" is for operator and "$$" is for vars. In addition, the translator 
will be triggered several times, each round for a legal ~= (a currentPK is not 
the same one of a previous PKs sets). You know, we need to increment the vars 
counter in each round after we generate new vars for the substituting branches' 
vars. As well as the following line 356, we can thus generate identical vars 
for all rounds of var generation requests.


Line 329: // Step3.3. the suffix 0-3 is used for identifying the 
different level of variable references.
> Can you present an example? different levels?
Nothing special, it is just for the anchor "#LEFT_1" in line 90 of the AQL 
template, to generate the vars for this anchor.


Line 356: 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

2016-11-07 Thread Wenhai Li (Code Review)
Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 10:

(15 comments)

@Taewoo,

Sorry, I did not know you cann't see the comments without publishing. :)

OK, published. Maybe we can talk with some detail with running example. You 
know, for those specified detail, these pessimistic methods can always forbid 
the exception of the dynamic variables generation.

https://asterix-gerrit.ics.uci.edu/#/c/1076/10/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
File 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java:

Line 83: // Step1: Initial the host embedding aql to substitute the fuzzy 
euqal condition such as $r.a ~= $s.b
> What does "Initial the host embedding aql" mean?
This is grammatical mistake. Here, initial = initialize. This is a substituted 
template to ingest the leaves of the fuzzyjoin and translate them into the 
equal fuzzyjoin subplan. It means that the following template AQLPLUS will be 
embedded regardless the token functions.


Line 88: + "((#LEFT_%d_0), " + "  (join((#RIGHT_%d_0), "
> %d change is necessary because?
I our old version, we always duplicate the vars with a 
LogicalOperatorDeepCopyVaraiblesVisitor other than the newest 
LogicalOperatorDeepCopyWithNewVaraiblesVisitor, we will get duplicated vars if 
not differentiate the vars like this. Now, I "have removed" this var in the 
version 12 of this patch.

Even by this, the following remarks are also useful for the function of this 
variable.
As will be replaced by the nBranches, this %d formatter is used to 
differentiate the VarIdentifier (line 311-338 in Step 3.1-3.3) in the 
multi-fuzzyjoin context, where additional leaf vars will be introduced into the 
translator. We need to different the leaf identifier once this rule is 
triggered multiple times in the multi-fuzzyjoin. Otherwise, we will get the 
same var identifier for the different fuzzyjoin branches. For example,
for $r in dataset DBLP
for $s in dataset CSX\
for $t in dataset ACM
where word-tokens($r.title) ~= word-tokens($s.title) and word-tokens($r.title) 
~= word-tokens($t.title)
return {"rid":$r.id, "sid":$s.id, "tid":$t.id}
, we will get a plan abstracted as (word-token($r.title) fj 
word-tokens($s.title)) => (word-token($r.title) fj word-tokens($t.title)
. The var substitution will make the three identical leaves different.


Line 201: VariableUtilities.getLiveVariables(rightInputOp, liveVars);
> What do we want to achieve for this change (line 201 - 220)? Can you put mo
This is a segment especially for the SELECT based on fuzzy join on the same 
table pair. Let take an example for this:
for $d in dataset DBLP
for $t in dataset CSX
where word-tokens($d.title) ~= word-tokens($t.title)
and word-tokens($d.authors) ~= word-tokens($t.authors)
return {"did": $d.id, "tid": $t.id}
Actually, these two fuzzyjoin are placed on the same table pair. Of course we 
can fire two fuzzyjoins consequently. Nevertheless, a cheaper method is to 
setup a fuzzyjoin on the first condition and conduct a SELECT towards its 
result. Is well known that the fuzzy join will generally propagate the input 
cardinalities of the join  branches.

Here, pk are appended onto the static previousPK list after each round of 
fuzzyjoin are ingested. We can differentiate whether a same table-pair has been 
translated.


Line 250: if (constVal.getObject() instanceof AFloat) {
> instanceOf can be replaced by using constVal.getObject() and BuiltinType.
Done


Line 253: simThreshold = 
FuzzyUtils.getSimThreshold(metadataProvider);
> Do we have this case?
By default, without user input?


Line 342: for (int i = 1; i < 4; i++) {
> Why this is fixed number? i < 4?
right side will be used as the token order source, right record order and 
surrogate join. Please refer to the template AQLPLUS.


Line 378: // Step 5. Bind the plan to the father op referred by the 
below opRef.
> Can we put an example here? "the father op referred by the below opRef." is
Also refer an example.
Here, suppose the fuzzyjoin are word-tokens(DBLP.title) fj 
word-tokens(CSX.title) return {DBLP.id, CSX.id}. The father means the operator 
depending on fj, i.e. return {DBLP.id, CSX.id}. In general, this "parent" is 
the upstream operator that referred to fj in the original plan.


Line 408: private void setConditionForLeftOuterJoin(LeftOuterJoinOperator 
topJoin, Mutable expRef) {
> In the above, can we put some comments? What does this function do?
Done


Line 441: Mutable expRefRet = 
getSimilarityExpression(arg);
> Is this change necessary?
It is up to the reviewer. :)


Line 451: private List 
findPrimaryKeysInSubplan(Collection liveVars,
> In the above, can we put some 

Change in asterixdb[master]: Applied the multiway fuzzyjoin based on the prefix-based joi...

2016-11-07 Thread Wenhai Li (Code Review)
Wenhai Li has posted comments on this change.

Change subject: Applied the multiway fuzzyjoin based on the prefix-based join 
and the selectFuzzyJoin testCases.
..


Patch Set 26:

@Taewoo,

OK, published. Maybe we can talk with some detail with running example. You 
know, for those specified detail, these pessimistic methods can always forbid 
the exception of the dynamic variables generation.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1076
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8736f104905eeda763d39709e002c2b9629278cc
Gerrit-PatchSet: 26
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Wenhai Li 
Gerrit-Reviewer: Chen Li 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Wenhai Li 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..


Patch Set 2: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1100/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1346
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Michael Blow 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..


Patch Set 2:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1100/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1346
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Michael Blow 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..


Patch Set 2:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1346
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Michael Blow 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Jenkins (Code Review)
Jenkins has posted comments on this change.

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3262/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1346
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Michael Blow 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No


Change in asterixdb[master]: ASTERIXDB-1570: "Unable to find free page in buffer cache af...

2016-11-07 Thread Michael Blow (Code Review)
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1346

to look at the new patch set (#2).

Change subject: ASTERIXDB-1570: "Unable to find free page in buffer cache after 
1000 cycles"
..

ASTERIXDB-1570: "Unable to find free page in buffer cache after 1000 cycles"

Increase test buffer cache size to reliably accomodate 20M values

Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
---
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
M 
asterixdb/asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java
M 
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
4 files changed, 12 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/46/1346/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1346
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Michael Blow 
Gerrit-Reviewer: Jenkins 


Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

2016-11-07 Thread Wail Alkowaileet (Code Review)
Wail Alkowaileet has posted comments on this change.

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..


Patch Set 4:

For some reason, I cannot inline my comment. 
I will follow this format:
{line# : code
comment}

220 : if (curRecType.isClosedField(curFNames[iter1])) 
should that check if it's optional instead of closed. Because a field can be 
optional and that allows null.

Probably we can get rid of the first part (213 - 230). And make:
recType = curRecType == null ? ARecordType.FULLY_OPEN_RECORD_TYPE : curRecType;

236 : if (obj.isNull(attrName) || obj.length() == 0) 
is obj.length() ever going to equal zero? obj is the parent of attrName.

258: if (writeRecord == true) 
no need for == true

Great work.. Thanks!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No