Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Chris Hillery has submitted this change and it was merged.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT,
HyracksVirtualCluster.
..
ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
NCServiceExecutionIT runs all execution tests against a local cluster
managed by the NCService deployment framework.
HyracksVirtualCluster offers programmatic NCService deployment
control along with improved HyracksNCProcess/HyracksCCProcess.
Further fixes and improvements:
1. Fix handling of iodevices/storagedir (ASTERIXDB-1482)
2. Proper handling of [nc] default section in all cases
3. Ensure asterixnc, etc. scripts are executable
4. Consolidate Ini handling
5. Pruned some dead code, including VirtualClusterDriver
6. A bit of refactoring and extended commenting
Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Reviewed-on: https://asterix-gerrit.ics.uci.edu/958
Tested-by: Jenkins
Reviewed-by: Ian Maxon
---
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
M asterixdb/asterix-server/pom.xml
A
asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
A
asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
A
asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
M hyracks-fullstack/hyracks/hyracks-server/pom.xml
D
hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
M
hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
M
hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
M
hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
A
hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
M
hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
22 files changed, 536 insertions(+), 369 deletions(-)
Approvals:
Ian Maxon: Looks good to me, approved
Jenkins: Verified
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
index 9a8fba4..677fc78 100644
---
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
+++
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
@@ -39,7 +39,7 @@
}
public ClusterPartition getMetadataPartition() {
-return accessor.getMetadataPartiton();
+return accessor.getMetadataPartition();
}
public Map getStores() {
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
index 507a393..7309f0c 100644
---
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
+++
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
@@ -18,6 +18,7 @@
*/
package org.apache.asterix.common.config;
+import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
@@ -53,10 +54,12 @@
private final List nodeNames = new ArrayList<>();;
privat
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Ian Maxon has posted comments on this change. Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster. .. Patch Set 3: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/958 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Chris Hillery Gerrit-Reviewer: Chris Hillery Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Chris Hillery has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT,
HyracksVirtualCluster.
..
Patch Set 3:
(3 comments)
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/pom.xml
File asterixdb/asterix-server/pom.xml:
Line 127:
> What's this plugin about?
I copied these plugin settings from hyracks-server, where it's doing the same
thing: making sure that the appassembler-generated .sh scripts are executable.
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
File
asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java:
Line 135: LOGGER.info("Sleeping while cluster comes online...");
> Is there a better way to do this? Maybe trying to fetch against some port o
I'm not sure.. I suppose it could poll the rest/nodes Hyracks endpoint until
the expected number of NCs are connected?
https://asterix-gerrit.ics.uci.edu/#/c/958/3/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
File
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java:
Line 155
> Why remove this?
Because it no longer serves any purpose, with VirtualClusterDriver removed. It
was always an ugly hack that risked being out of sync, and with the NCService
model I'm trying to minimize the use of command-line arguments anyway.
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery
Gerrit-Reviewer: Chris Hillery
Gerrit-Reviewer: Ian Maxon
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Till Westmann
Gerrit-HasComments: Yes
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Ian Maxon has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT,
HyracksVirtualCluster.
..
Patch Set 3: Code-Review+1
(3 comments)
A few thoughts/questions
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/pom.xml
File asterixdb/asterix-server/pom.xml:
Line 127:
What's this plugin about?
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
File
asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java:
Line 135: LOGGER.info("Sleeping while cluster comes online...");
Is there a better way to do this? Maybe trying to fetch against some port on
the CC until it returns or a number of retries is reached?
https://asterix-gerrit.ics.uci.edu/#/c/958/3/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
File
hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java:
Line 155
Why remove this?
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery
Gerrit-Reviewer: Chris Hillery
Gerrit-Reviewer: Ian Maxon
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Till Westmann
Gerrit-HasComments: Yes
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster. .. Patch Set 3: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/958 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Chris Hillery Gerrit-Reviewer: Chris Hillery Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster. .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/1735/ -- To view, visit https://asterix-gerrit.ics.uci.edu/958 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Chris Hillery Gerrit-Reviewer: Chris Hillery Gerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/958 to look at the new patch set (#3). Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster. .. ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster. NCServiceExecutionIT runs all execution tests against a local cluster managed by the NCService deployment framework. HyracksVirtualCluster offers programmatic NCService deployment control along with improved HyracksNCProcess/HyracksCCProcess. Further fixes and improvements: 1. Fix handling of iodevices/storagedir (ASTERIXDB-1482) 2. Proper handling of [nc] default section in all cases 3. Ensure asterixnc, etc. scripts are executable 4. Consolidate Ini handling 5. Pruned some dead code, including VirtualClusterDriver 6. A bit of refactoring and extended commenting Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6 --- M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java M asterixdb/asterix-server/pom.xml A asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java M hyracks-fullstack/hyracks/hyracks-server/pom.xml D hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java A hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java M hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java M hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties 22 files changed, 536 insertions(+), 369 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/58/958/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/958 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Chris Hillery Gerrit-Reviewer: Chris Hillery Gerrit-Reviewer: Jenkins
