[1/2] git commit: SLIDER-181. Storm package updates
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-350-windows e9a353c1a - bfdec5897
SLIDER-181. Storm package updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/38c816eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/38c816eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/38c816eb
Branch: refs/heads/feature/SLIDER-350-windows
Commit: 38c816ebbd3c883308bbe36fc00846d4ff4dac7b
Parents: e9a353c
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Sep 3 13:15:20 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Sep 3 13:46:02 2014 -0700
--
app-packages/storm/README.txt | 19 ++---
app-packages/storm/appConfig.json | 10 +--
app-packages/storm/metainfo.xml | 2 +-
...pache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE | 16
app-packages/storm/pom.xml | 90
app-packages/storm/src/assembly/storm.xml | 68 +++
6 files changed, 172 insertions(+), 33 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38c816eb/app-packages/storm/README.txt
--
diff --git a/app-packages/storm/README.txt b/app-packages/storm/README.txt
index d4df85f..49adfce 100644
--- a/app-packages/storm/README.txt
+++ b/app-packages/storm/README.txt
@@ -17,19 +17,16 @@
How to create a Slider app package for Storm?
-To create the app package you will need the Storm tarball copied to a specific
location.
-Various configurations provided in this sample are customized for
apache-storm-0.9.3.2.2.0.0-578.tar.gz.
-So if you use a different version you may need to edit a few config values.
+To create the app package you will need the Storm tarball and invoke mvn
command
+with appropriate parameters.
-Replace the placeholder tarball for Storm.
- cp ~/Downloads/apache-storm-0.9.3.2.2.0.0-578.tar.gz package/files/
- rm package/files/apache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE
+Command:
+mvn clean package -Pstorm-app-package -Dpkg.version=version
+ -Dpkg.name=file name of app tarball -Dpkg.src=folder location where the
pkg is available
-Create a zip package at the root of the package (slider
enlistment/app-packages/storm/)
- zip -r apache-storm-0.9.3.zip .
-
-Verify the content using
- unzip -l $@ apache-storm-0.9.3.zip
+Example:
+mvn clean package -Pstorm-app-package -Dpkg.version=0.9.3.2.2.0.0-578
+ -Dpkg.name=apache-storm-0.9.3.2.2.0.0-578.tar.gz
-Dpkg.src=/Users/user1/Downloads
While appConfig.json and resources.json are not required for the package they
work
well as the default configuration for Slider apps. So its advisable that when
you
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38c816eb/app-packages/storm/appConfig.json
--
diff --git a/app-packages/storm/appConfig.json
b/app-packages/storm/appConfig.json
index b5fe436..df28e8f 100644
--- a/app-packages/storm/appConfig.json
+++ b/app-packages/storm/appConfig.json
@@ -3,13 +3,13 @@
metadata: {
},
global: {
-application.def: apache-storm-0.9.3.zip,
+application.def: slider-storm-app-package-${pkg.version}.zip,
java_home: /usr/jdk64/jdk1.7.0_45,
create.default.zookeeper.node: true,
system_configs: core-site,
site.global.app_user: yarn,
-site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version},
site.global.user_group: hadoop,
site.global.ganglia_server_host: ${NN_HOST},
site.global.ganglia_server_id: Application2,
@@ -22,8 +22,8 @@
site.storm-site.storm.local.dir: ${AGENT_WORK_ROOT}/app/tmp/storm,
site.storm-site.transactional.zookeeper.root: /transactional,
site.storm-site.storm.zookeeper.port: 2181,
-site.storm-site.nimbus.childopts: -Xmx1024m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM,
-site.storm-site.worker.childopts: -Xmx768m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM,
+site.storm-site.nimbus.childopts:
[2/2] git commit: SLIDER-350 switch to hadoop 3.0.0 for testing
SLIDER-350 switch to hadoop 3.0.0 for testing Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/bfdec589 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/bfdec589 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/bfdec589 Branch: refs/heads/feature/SLIDER-350-windows Commit: bfdec5897997511cef405885bd940da01a5189e7 Parents: 38c816e Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 12:59:28 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 12:59:28 2014 +0100 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bfdec589/pom.xml -- diff --git a/pom.xml b/pom.xml index 8292073..454918c 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,7 @@ !-- core artifacts -- -hadoop.version2.6.0-SNAPSHOT/hadoop.version +hadoop.version3.0.0-SNAPSHOT/hadoop.version hbase.version0.98.4-hadoop2/hbase.version accumulo.version1.6.1-SNAPSHOT/accumulo.version
git commit: SLIDER-350 add strict test for windows utils being valid
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-350-windows bfdec5897 - 2e3c4c81b
SLIDER-350 add strict test for windows utils being valid
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2e3c4c81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2e3c4c81
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2e3c4c81
Branch: refs/heads/feature/SLIDER-350-windows
Commit: 2e3c4c81b799cbf6d06e394c2b17f8f161f68734
Parents: bfdec58
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 13:46:59 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 13:46:59 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 50 +++-
.../common/tools/TestWindowsSupport.groovy | 30 ++--
2 files changed, 76 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2e3c4c81/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index a864878..eb214db 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -37,6 +37,7 @@ import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Shell;
import org.apache.hadoop.util.VersionInfo;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.Container;
@@ -63,6 +64,7 @@ import org.slf4j.LoggerFactory;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
@@ -104,8 +106,9 @@ public final class SliderUtils {
public static final String JAVA_SECURITY_KRB5_REALM =
java.security.krb5.realm;
public static final String JAVA_SECURITY_KRB5_KDC = java.security.krb5.kdc;
+ public static final String WINUTILS = WINUTILS.EXE;
+
-
private SliderUtils() {
}
@@ -1499,4 +1502,49 @@ public final class SliderUtils {
return is;
}
+
+ /**
+ * Strictly verify that windows utils is present.
+ * Checks go as far as opening the file and looking for
+ * the headers.
+ * @throws IOException on any problem reading the file
+ * @throws FileNotFoundException if the file is not considered valid
+ */
+ public static void verifyWinUtilsValid() throws IOException {
+if (!Shell.WINDOWS) {
+ return;
+}
+String winUtilsPath = Shell.getWinUtilsPath();
+if (winUtilsPath == null) {
+ throw new FileNotFoundException(WINUTILS + not found on Path : +
+ System.getenv(Path));
+}
+File winUtils = new File(winUtilsPath);
+if (!winUtils.isFile()) {
+ throw new FileNotFoundException(WINUTILS
+ + at + winUtilsPath
+ + is not a file);
+
+}
+if (winUtils.length() 0x100) {
+ throw new FileNotFoundException(WINUTILS
++ at + winUtilsPath
++ is too short to be an executable);
+}
+// now read two bytes and verify the header.
+FileReader reader = null;
+try {
+ int[] header = new int[2];
+ reader = new FileReader(winUtilsPath);
+ header[0] = reader.read();
+ header[1] = reader.read();
+ if (header[0] != 'M' || header[1] != 'Z') {
+throw new FileNotFoundException(WINUTILS
+ + at + winUtilsPath
+ + is not a windows executable file);
+ }
+} finally {
+ IOUtils.closeStream(reader);
+}
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2e3c4c81/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index 7f5edf3..84356ea 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -27,14 +27,16 @@ import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.FileSystem as HadoopFS
[1/2] git commit: SLIDER-350 use gawk as awk implementation for windows; add checks for that and xargs on the command line
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-350-windows 690793a4f - 93c3c770f
SLIDER-350 use gawk as awk implementation for windows; add checks for that and
xargs on the command line
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8f9caa9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8f9caa9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8f9caa9b
Branch: refs/heads/feature/SLIDER-350-windows
Commit: 8f9caa9ba8ba5a09df794096d0096356535dce38
Parents: 690793a
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:01:30 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:01:30 2014 +0100
--
.../org/apache/slider/common/tools/TestWindowsSupport.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8f9caa9b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index d18fe76..61ede62 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -155,7 +155,7 @@ class TestWindowsSupport extends SliderTestBase {
* @param commands
* @return
*/
- public ForkedProcessService exec(ArrayListString commands) {
+ public ForkedProcessService exec(ListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
methodName.methodName,
[2/2] git commit: SLIDER-350 windows support
SLIDER-350 windows support
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/93c3c770
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/93c3c770
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/93c3c770
Branch: refs/heads/feature/SLIDER-350-windows
Commit: 93c3c770fed878895e4fb67ac17cbbb098cb4fb3
Parents: 8f9caa9
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:05:38 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:05:38 2014 +0100
--
.../common/tools/TestWindowsSupport.groovy | 22 +++-
1 file changed, 17 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/93c3c770/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index 61ede62..6ae683a 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -109,13 +109,13 @@ class TestWindowsSupport extends SliderTestBase {
@Test
public void testHasGawkInstalled() throws Throwable {
assume(Shell.WINDOWS, not windows)
-assert 0 == exec([gawk, --version])
+exec(0, [gawk, --version])
}
@Test
public void testHasXargsInstalled() throws Throwable {
assume(Shell.WINDOWS, not windows)
-assert 0 == exec([xargs, --version])
+exec(0, [xargs, --version])
}
@@ -145,16 +145,28 @@ class TestWindowsSupport extends SliderTestBase {
File winUtils = new File(winUtilsPath)
log.debug(Winutils is at $winUtils))
-ForkedProcessService process = exec([winUtilsPath, systeminfo])
-assert 0 == process.exitCode
+exec(0, [winUtilsPath, systeminfo])
}
/**
* Exec a set of commands, wait a few seconds for it to finish.
+ * @param status code
* @param commands
- * @return
+ * @return the process
*/
+ public ForkedProcessService exec(int status, ListString commands) {
+ForkedProcessService process = exec(commands)
+assert status == process.exitCode
+return process
+ }
+
+ /**
+ * Exec a set of commands, wait a few seconds for it to finish.
+ * @param commands
+ * @return
+ */
+
public ForkedProcessService exec(ListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
git commit: SLIDER-377 switch to branch 2.6 to see if failures arise
Repository: incubator-slider Updated Branches: refs/heads/feature/SLIDER-350-windows 93c3c770f - 877a01609 SLIDER-377 switch to branch 2.6 to see if failures arise Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/877a0160 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/877a0160 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/877a0160 Branch: refs/heads/feature/SLIDER-350-windows Commit: 877a01609748546a021d5477b7cd52f7f1536c65 Parents: 93c3c77 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 14:11:29 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 14:11:29 2014 +0100 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/877a0160/pom.xml -- diff --git a/pom.xml b/pom.xml index 454918c..8292073 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,7 @@ !-- core artifacts -- -hadoop.version3.0.0-SNAPSHOT/hadoop.version +hadoop.version2.6.0-SNAPSHOT/hadoop.version hbase.version0.98.4-hadoop2/hbase.version accumulo.version1.6.1-SNAPSHOT/accumulo.version
[1/7] git commit: SLIDER-350 switch to hadoop 3.0.0 for testing
Repository: incubator-slider Updated Branches: refs/heads/develop 38c816ebb - 29a40f509 SLIDER-350 switch to hadoop 3.0.0 for testing Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/bfdec589 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/bfdec589 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/bfdec589 Branch: refs/heads/develop Commit: bfdec5897997511cef405885bd940da01a5189e7 Parents: 38c816e Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 12:59:28 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 12:59:28 2014 +0100 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bfdec589/pom.xml -- diff --git a/pom.xml b/pom.xml index 8292073..454918c 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,7 @@ !-- core artifacts -- -hadoop.version2.6.0-SNAPSHOT/hadoop.version +hadoop.version3.0.0-SNAPSHOT/hadoop.version hbase.version0.98.4-hadoop2/hbase.version accumulo.version1.6.1-SNAPSHOT/accumulo.version
[4/7] git commit: SLIDER-350 use gawk as awk implementation for windows; add checks for that and xargs on the command line
SLIDER-350 use gawk as awk implementation for windows; add checks for that and
xargs on the command line
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8f9caa9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8f9caa9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8f9caa9b
Branch: refs/heads/develop
Commit: 8f9caa9ba8ba5a09df794096d0096356535dce38
Parents: 690793a
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:01:30 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:01:30 2014 +0100
--
.../org/apache/slider/common/tools/TestWindowsSupport.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8f9caa9b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index d18fe76..61ede62 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -155,7 +155,7 @@ class TestWindowsSupport extends SliderTestBase {
* @param commands
* @return
*/
- public ForkedProcessService exec(ArrayListString commands) {
+ public ForkedProcessService exec(ListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
methodName.methodName,
[6/7] git commit: SLIDER-377 switch to branch 2.6 to see if failures arise
SLIDER-377 switch to branch 2.6 to see if failures arise Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/877a0160 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/877a0160 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/877a0160 Branch: refs/heads/develop Commit: 877a01609748546a021d5477b7cd52f7f1536c65 Parents: 93c3c77 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 14:11:29 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 14:11:29 2014 +0100 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/877a0160/pom.xml -- diff --git a/pom.xml b/pom.xml index 454918c..8292073 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,7 @@ !-- core artifacts -- -hadoop.version3.0.0-SNAPSHOT/hadoop.version +hadoop.version2.6.0-SNAPSHOT/hadoop.version hbase.version0.98.4-hadoop2/hbase.version accumulo.version1.6.1-SNAPSHOT/accumulo.version
[5/7] git commit: SLIDER-350 windows support
SLIDER-350 windows support
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/93c3c770
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/93c3c770
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/93c3c770
Branch: refs/heads/develop
Commit: 93c3c770fed878895e4fb67ac17cbbb098cb4fb3
Parents: 8f9caa9
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:05:38 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:05:38 2014 +0100
--
.../common/tools/TestWindowsSupport.groovy | 22 +++-
1 file changed, 17 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/93c3c770/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index 61ede62..6ae683a 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -109,13 +109,13 @@ class TestWindowsSupport extends SliderTestBase {
@Test
public void testHasGawkInstalled() throws Throwable {
assume(Shell.WINDOWS, not windows)
-assert 0 == exec([gawk, --version])
+exec(0, [gawk, --version])
}
@Test
public void testHasXargsInstalled() throws Throwable {
assume(Shell.WINDOWS, not windows)
-assert 0 == exec([xargs, --version])
+exec(0, [xargs, --version])
}
@@ -145,16 +145,28 @@ class TestWindowsSupport extends SliderTestBase {
File winUtils = new File(winUtilsPath)
log.debug(Winutils is at $winUtils))
-ForkedProcessService process = exec([winUtilsPath, systeminfo])
-assert 0 == process.exitCode
+exec(0, [winUtilsPath, systeminfo])
}
/**
* Exec a set of commands, wait a few seconds for it to finish.
+ * @param status code
* @param commands
- * @return
+ * @return the process
*/
+ public ForkedProcessService exec(int status, ListString commands) {
+ForkedProcessService process = exec(commands)
+assert status == process.exitCode
+return process
+ }
+
+ /**
+ * Exec a set of commands, wait a few seconds for it to finish.
+ * @param commands
+ * @return
+ */
+
public ForkedProcessService exec(ListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
git commit: SLIDER-146 added sample appConfig that enables kerberos for Accumulo
Repository: incubator-slider
Updated Branches:
refs/heads/develop 29a40f509 - 6320f6186
SLIDER-146 added sample appConfig that enables kerberos for Accumulo
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6320f618
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6320f618
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6320f618
Branch: refs/heads/develop
Commit: 6320f6186dec56f68afbbd638f5ddc0e7dc04bff
Parents: 29a40f5
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Thu Sep 4 07:15:22 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Thu Sep 4 07:15:22 2014 -0700
--
.../src/test/resources/appConfig_kerberos.json | 55
1 file changed, 55 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6320f618/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
--
diff --git a/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
b/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
new file mode 100644
index 000..c9684d4
--- /dev/null
+++ b/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
@@ -0,0 +1,55 @@
+{
+ schema: http://example.org/specification/v2.0.0;,
+ metadata: {
+ },
+ global: {
+application.def: ${app.package.name}.zip,
+java_home: /usr/lib/jvm/java,
+site.global.app_user: accumulo,
+site.global.app_log_dir: ${AGENT_LOG_ROOT},
+site.global.app_pid_dir: ${AGENT_WORK_ROOT}/app/run,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/accumulo-${accumulo.version},
+site.global.tserver_heapsize: 256m,
+site.global.master_heapsize: 128m,
+site.global.monitor_heapsize: 64m,
+site.global.gc_heapsize: 64m,
+site.global.other_heapsize: 128m,
+site.global.hadoop_prefix: /usr/lib/hadoop,
+site.global.hadoop_conf_dir: /etc/hadoop/conf,
+site.global.zookeeper_home: /usr/lib/zookeeper,
+site.global.accumulo_instance_name: ${USER}-${CLUSTER_NAME},
+site.global.accumulo_root_password: NOT_USED,
+site.global.user_group: accumulo,
+site.global.security_enabled: false,
+site.global.ssl_cert_dir: ssl,
+site.global.monitor_protocol: http,
+site.accumulo-site.instance.volumes: ${DEFAULT_DATA_DIR}/data,
+site.accumulo-site.instance.zookeeper.host: ${ZK_HOST},
+site.accumulo-site.instance.security.authenticator:
org.apache.slider.accumulo.CustomAuthenticator,
+site.accumulo-site.general.security.credential.provider.paths:
jceks://hdfs/user/${USER}/accumulo-${CLUSTER_NAME}.jceks,
+site.accumulo-site.general.kerberos.keytab:
/etc/security/keytabs/accumulo.service.keytab,
+site.accumulo-site.general.kerberos.principal:
accumulo/_h...@example.com,
+site.accumulo-site.tserver.memory.maps.native.enabled: false,
+site.accumulo-site.tserver.memory.maps.max: 80M,
+site.accumulo-site.tserver.cache.data.size: 7M,
+site.accumulo-site.tserver.cache.index.size: 20M,
+site.accumulo-site.trace.user: root,
+site.accumulo-site.tserver.sort.buffer.size: 50M,
+site.accumulo-site.tserver.walog.max.size: 40M,
+site.accumulo-site.master.port.client: 0,
+site.accumulo-site.trace.port.client: 0,
+site.accumulo-site.tserver.port.client: 0,
+site.accumulo-site.gc.port.client: 0,
+site.accumulo-site.monitor.port.client:
${ACCUMULO_MONITOR.ALLOCATED_PORT},
+site.accumulo-site.monitor.port.log4j: 0,
+site.accumulo-site.general.classpaths:
$ACCUMULO_HOME/lib/accumulo-server.jar,\n$ACCUMULO_HOME/lib/accumulo-core.jar,\n$ACCUMULO_HOME/lib/accumulo-start.jar,\n$ACCUMULO_HOME/lib/accumulo-fate.jar,\n$ACCUMULO_HOME/lib/accumulo-proxy.jar,\n$ACCUMULO_HOME/lib/[^.].*.jar,\n$ZOOKEEPER_HOME/zookeeper[^.].*.jar,\n$HADOOP_CONF_DIR,\n$HADOOP_PREFIX/[^.].*.jar,\n$HADOOP_PREFIX/lib/[^.].*.jar,\n$HADOOP_PREFIX/share/hadoop/common/.*.jar,\n$HADOOP_PREFIX/share/hadoop/common/lib/.*.jar,\n$HADOOP_PREFIX/share/hadoop/hdfs/.*.jar,\n$HADOOP_PREFIX/share/hadoop/mapreduce/.*.jar,\n$HADOOP_PREFIX/share/hadoop/yarn/.*.jar,\n/usr/lib/hadoop/.*.jar,\n/usr/lib/hadoop/lib/.*.jar,\n/usr/lib/hadoop-hdfs/.*.jar,\n/usr/lib/hadoop-mapreduce/.*.jar,\n/usr/lib/hadoop-yarn/.*.jar,
+ },
+ credentials: {
+jceks://hdfs/user/${USER}/accumulo-${CLUSTER_NAME}.jceks:
[root.initial.password, instance.secret, trace.token.property.password]
+ },
+ components: {
+slider-appmaster: {
+ jvm.heapsize: 256M
+}
+ }
+}
git commit: SLIDER-394 slider client and AM services to fail-fast if the hadoop dependencies are missing
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-394_env_check [created] 0f67b1042
SLIDER-394 slider client and AM services to fail-fast if the hadoop
dependencies are missing
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0f67b104
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0f67b104
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0f67b104
Branch: refs/heads/feature/SLIDER-394_env_check
Commit: 0f67b10424cb807aebb3d2b15c3b24149f4ae70a
Parents: 29a40f5
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:28:20 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:28:20 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 555 +--
.../services/utility/EndOfServiceWaiter.java| 84 +++
.../services/workflow/ForkedProcessService.java | 21 +-
.../services/workflow/LongLivedProcess.java | 38 +-
.../tools/TestExecutionEnvironment.groovy | 42 ++
.../common/tools/TestWindowsSupport.groovy | 4 +-
.../services/workflow/EndOfServiceWaiter.java | 56 --
.../TestWorkflowForkedProcessService.java | 4 +-
8 files changed, 570 insertions(+), 234 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0f67b104/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index eb214db..44fe4fd 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -56,7 +56,10 @@ import org.apache.slider.core.exceptions.ErrorStrings;
import org.apache.slider.core.exceptions.MissingArgException;
import org.apache.slider.core.exceptions.SliderException;
import org.apache.slider.core.launch.ClasspathConstructor;
+import org.apache.slider.core.main.LauncherExitCodes;
+import org.apache.slider.server.services.utility.EndOfServiceWaiter;
import org.apache.slider.server.services.utility.PatternValidator;
+import org.apache.slider.server.services.workflow.ForkedProcessService;
import org.apache.zookeeper.server.util.KerberosUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -67,6 +70,7 @@ import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
+import java.io.InterruptedIOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.net.InetSocketAddress;
@@ -88,6 +92,7 @@ import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
import java.util.TreeSet;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
/**
@@ -102,12 +107,24 @@ public final class SliderUtils {
* turned on (prevents re-entrancy)
*/
private static final AtomicBoolean processSecurityAlreadyInitialized =
-new AtomicBoolean(false);
+ new AtomicBoolean(false);
public static final String JAVA_SECURITY_KRB5_REALM =
java.security.krb5.realm;
public static final String JAVA_SECURITY_KRB5_KDC = java.security.krb5.kdc;
+
+ /**
+ * Winutils
+ */
public static final String WINUTILS = WINUTILS.EXE;
+ /**
+ * name of openssl program
+ */
+ public static final String OPENSSL = openssl;
+ /**
+ * name of python program
+ */
+ public static final String PYTHON = python;
private SliderUtils() {
}
@@ -131,7 +148,8 @@ public final class SliderUtils {
* @param msg the message to be shown in exception
*/
@SuppressWarnings(ResultOfMethodCallIgnored)
- private static void validateNumber(String num, String msg) throws
BadConfigException {
+ private static void validateNumber(String num, String msg) throws
+ BadConfigException {
try {
Integer.parseInt(num);
} catch (NumberFormatException nfe) {
@@ -145,15 +163,16 @@ public final class SliderUtils {
* @param heapsize
* @return heapsize in MB
*/
- public static String translateTrailingHeapUnit(String heapsize) throws
BadConfigException {
+ public static String translateTrailingHeapUnit(String heapsize) throws
+ BadConfigException {
String errMsg = Bad heapsize: ;
if (heapsize.endsWith(m) || heapsize.endsWith(M)) {
- String num = heapsize.substring(0, heapsize.length()-1);
+ String num = heapsize.substring(0, heapsize.length() - 1);
validateNumber(num, errMsg);
return num;
}
if (heapsize.endsWith(g) || heapsize.endsWith(G)) {
- String
git commit: SLIDER-394 slider client and AM services to fail-fast if the hadoop dependencies are missing
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-394_env_check 0f67b1042 - 8034012f2
SLIDER-394 slider client and AM services to fail-fast if the hadoop
dependencies are missing
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8034012f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8034012f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8034012f
Branch: refs/heads/feature/SLIDER-394_env_check
Commit: 8034012f2f0282c9dbab814a282ab47abc175204
Parents: 0f67b10
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:37:05 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:37:05 2014 +0100
--
.../slider/server/appmaster/SliderAppMaster.java| 3 +++
.../server/services/workflow/LongLivedProcess.java | 16 +++-
2 files changed, 18 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8034012f/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
index 9ccb382..49c7792 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
@@ -400,6 +400,9 @@ public class SliderAppMaster extends
AbstractSliderLaunchedService
//look at settings of Hadoop Auth, to pick up a problem seen once
checkAndWarnForAuthTokenProblems();
+
+// validate server env
+SliderUtils.validateSliderServerEnvironment(log);
executorService = new
WorkflowExecutorServiceExecutorService(AmExecutor,
Executors.newCachedThreadPool(
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8034012f/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
b/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
index 6f83811..a0df2c9 100644
---
a/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
+++
b/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
@@ -24,6 +24,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.BufferedReader;
+import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
@@ -292,7 +293,20 @@ public class LongLivedProcess implements Runnable {
if (LOG.isDebugEnabled()) {
LOG.debug(Spawning process:\n + describeBuilder());
}
-process = processBuilder.start();
+try {
+ process = processBuilder.start();
+} catch (IOException e) {
+ // on windows, upconvert DOS error 2 from ::CreateProcess()
+ // to its real meaning: FileNotFound
+ if (e.toString().contains(CreateProcess error=2)) {
+FileNotFoundException fnfe =
+new FileNotFoundException(e.toString());
+fnfe.initCause(e);
+throw fnfe;
+ } else {
+throw e;
+ }
+}
return process;
}
git commit: SLIDER-377 AM checks for validity of server side; client does this if the CLI action says it should (default is yes)
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-394_env_check 8034012f2 - b63618039
SLIDER-377 AM checks for validity of server side; client does this if the CLI
action says it should (default is yes)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b6361803
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b6361803
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b6361803
Branch: refs/heads/feature/SLIDER-394_env_check
Commit: b636180392a719b1b663f77592c23b7b16a8f4d3
Parents: 8034012
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:53:35 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:53:35 2014 +0100
--
.../org/apache/slider/client/SliderClient.java| 18 ++
.../slider/common/params/AbstractActionArgs.java | 11 +++
.../slider/common/params/ActionHelpArgs.java | 14 --
.../slider/common/params/ActionVersionArgs.java | 12 +++-
.../apache/slider/common/tools/SliderUtils.java | 2 +-
.../apache/slider/core/build/InstanceBuilder.java | 5 +++--
.../slider/server/appmaster/SliderAppMaster.java | 2 +-
.../services/workflow/LongLivedProcess.java | 3 ++-
8 files changed, 55 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b6361803/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index f6d761e..f4040c4 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -46,6 +46,7 @@ import org.apache.slider.api.proto.Messages;
import org.apache.slider.common.Constants;
import org.apache.slider.common.SliderExitCodes;
import org.apache.slider.common.SliderKeys;
+import org.apache.slider.common.params.AbstractActionArgs;
import org.apache.slider.common.params.AbstractClusterBuildingActionArgs;
import org.apache.slider.common.params.ActionAMSuicideArgs;
import org.apache.slider.common.params.ActionCreateArgs;
@@ -291,6 +292,12 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
// choose the action
String action = serviceArgs.getAction();
+
+AbstractActionArgs coreAction = serviceArgs.getCoreAction();
+if (coreAction.getHadoopServicesRequired()) {
+ // validate the client
+ SliderUtils.validateSliderClientEnvironment(null);
+}
int exitCode = EXIT_SUCCESS;
String clusterName = serviceArgs.getClusterName();
// actions
@@ -613,7 +620,9 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
// verify that a live cluster isn't there
SliderUtils.validateClusterName(clustername);
verifyBindingsDefined();
-if (!liveClusterAllowed) verifyNoLiveClusters(clustername);
+if (!liveClusterAllowed) {
+ verifyNoLiveClusters(clustername);
+}
Configuration conf = getConfig();
String registryQuorum = lookupZKQuorum();
@@ -1169,7 +1178,7 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
*/
addConfOptionToCLI(commandLine,
config,
- DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
+ DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY);
}
// write out the path output
commandLine.addOutAndErrFiles(STDOUT_AM, STDERR_AM);
@@ -1263,10 +1272,11 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
*/
private void propagatePrincipals(Configuration config,
AggregateConf clusterSpec) {
-String dfsPrincipal = config.get(DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
+String dfsPrincipal = config.get(
+DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY);
if (dfsPrincipal != null) {
String siteDfsPrincipal = OptionKeys.SITE_XML_PREFIX +
-DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY;
+
DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY;
clusterSpec.getAppConfOperations().getGlobalOptions().putIfUnset(
siteDfsPrincipal,
dfsPrincipal);
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b6361803/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
--
diff --git
git commit: SLIDER-377 AM checks for validity of server side; client does this if the CLI action says it should (default is yes)
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-394_env_check b63618039 - 016427702
SLIDER-377 AM checks for validity of server side; client does this if the CLI
action says it should (default is yes)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/01642770
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/01642770
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/01642770
Branch: refs/heads/feature/SLIDER-394_env_check
Commit: 0164277025ce3f8ec0a53b4dbd1005967632262e
Parents: b636180
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:04:46 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:04:46 2014 +0100
--
.../apache/slider/providers/agent/AgentTestBase.groovy| 10 ++
.../minicluster/masterless/TestRoleOptPropagation.groovy | 4 +---
2 files changed, 11 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/01642770/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
index 6dee64f..2953381 100644
---
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
@@ -26,9 +26,11 @@ import org.apache.commons.compress.utils.IOUtils
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.apache.slider.client.SliderClient
import org.apache.slider.common.params.SliderActions
+import org.apache.slider.common.tools.SliderUtils
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.test.YarnZKMiniClusterTestBase
import org.junit.Before
+import org.junit.BeforeClass
import org.junit.Rule
import org.junit.rules.TemporaryFolder
@@ -47,6 +49,14 @@ public abstract class AgentTestBase extends
YarnZKMiniClusterTestBase {
@Rule
public TemporaryFolder folder = new TemporaryFolder();
+ /**
+ * Server side test: validate system env before launch
+ */
+ @BeforeClass
+ public static void checkSystem() {
+SliderUtils.validateSliderServerEnvironment(null)
+ }
+
public String app_def_pkg_path;
@Before
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/01642770/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
--
diff --git
a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
index cffde39..dd9b8e9 100644
---
a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
+++
b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
@@ -30,8 +30,6 @@ import
org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
import org.apache.slider.core.main.ServiceLauncher
import org.junit.Test
-import static HBaseKeys.PROVIDER_HBASE
-import static Arguments.ARG_PROVIDER
@CompileStatic
@Slf4j
@@ -54,7 +52,7 @@ class TestRoleOptPropagation extends HBaseMiniClusterTestBase
{
],
[
Arguments.ARG_COMP_OPT, HBaseKeys.ROLE_MASTER, ENV, 4,
- ARG_PROVIDER, PROVIDER_HBASE
+ Arguments.ARG_PROVIDER, HBaseKeys.PROVIDER_HBASE
],
true,
true,
git commit: SLIDER-377 AM-spawning tests do prelaunch validity checks
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-394_env_check 016427702 - 2d6f276d1
SLIDER-377 AM-spawning tests do prelaunch validity checks
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2d6f276d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2d6f276d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2d6f276d
Branch: refs/heads/feature/SLIDER-394_env_check
Commit: 2d6f276d16cc93648ccf7306eb94f5ac09bb5763
Parents: 0164277
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:09:12 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:09:12 2014 +0100
--
.../org/apache/slider/agent/AgentMiniClusterTestBase.groovy | 9 +
1 file changed, 9 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2d6f276d/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
index 4566d9d..7786c41 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
@@ -27,6 +27,7 @@ import org.apache.commons.io.FileUtils
import org.apache.slider.client.SliderClient
import org.apache.slider.common.SliderXMLConfKeysForTesting
import org.apache.slider.common.params.Arguments
+import org.apache.slider.common.tools.SliderUtils
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.providers.agent.AgentKeys
import org.apache.slider.test.YarnZKMiniClusterTestBase
@@ -47,6 +48,14 @@ extends YarnZKMiniClusterTestBase {
protected static MapString, String agentDefOptions
private static TemporaryFolder tempFolder = new TemporaryFolder();
+ /**
+ * Server side test: validate system env before launch
+ */
+ @BeforeClass
+ public static void checkSystem() {
+SliderUtils.validateSliderServerEnvironment(null)
+ }
+
@BeforeClass
public static void createSubConfFiles() {
[6/6] git commit: Merge branch 'feature/SLIDER-394_env_check' into develop
Merge branch 'feature/SLIDER-394_env_check' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/88d72ddf Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/88d72ddf Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/88d72ddf Branch: refs/heads/develop Commit: 88d72ddfb32977ff098b0329ae6f3faf2f7c2875 Parents: 6320f61 2d6f276 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 17:12:02 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 17:12:02 2014 +0100 -- .../org/apache/slider/client/SliderClient.java | 18 +- .../common/params/AbstractActionArgs.java | 11 + .../slider/common/params/ActionHelpArgs.java| 14 +- .../slider/common/params/ActionVersionArgs.java | 12 +- .../apache/slider/common/tools/SliderUtils.java | 557 +-- .../slider/core/build/InstanceBuilder.java | 5 +- .../server/appmaster/SliderAppMaster.java | 5 +- .../services/utility/EndOfServiceWaiter.java| 84 +++ .../services/workflow/ForkedProcessService.java | 21 +- .../services/workflow/LongLivedProcess.java | 57 +- .../agent/AgentMiniClusterTestBase.groovy | 9 + .../tools/TestExecutionEnvironment.groovy | 42 ++ .../common/tools/TestWindowsSupport.groovy | 4 +- .../slider/providers/agent/AgentTestBase.groovy | 10 + .../services/workflow/EndOfServiceWaiter.java | 56 -- .../TestWorkflowForkedProcessService.java | 4 +- .../masterless/TestRoleOptPropagation.groovy| 4 +- 17 files changed, 663 insertions(+), 250 deletions(-) --
[3/6] git commit: SLIDER-377 AM checks for validity of server side; client does this if the CLI action says it should (default is yes)
SLIDER-377 AM checks for validity of server side; client does this if the CLI
action says it should (default is yes)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b6361803
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b6361803
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b6361803
Branch: refs/heads/develop
Commit: b636180392a719b1b663f77592c23b7b16a8f4d3
Parents: 8034012
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:53:35 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:53:35 2014 +0100
--
.../org/apache/slider/client/SliderClient.java| 18 ++
.../slider/common/params/AbstractActionArgs.java | 11 +++
.../slider/common/params/ActionHelpArgs.java | 14 --
.../slider/common/params/ActionVersionArgs.java | 12 +++-
.../apache/slider/common/tools/SliderUtils.java | 2 +-
.../apache/slider/core/build/InstanceBuilder.java | 5 +++--
.../slider/server/appmaster/SliderAppMaster.java | 2 +-
.../services/workflow/LongLivedProcess.java | 3 ++-
8 files changed, 55 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b6361803/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index f6d761e..f4040c4 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -46,6 +46,7 @@ import org.apache.slider.api.proto.Messages;
import org.apache.slider.common.Constants;
import org.apache.slider.common.SliderExitCodes;
import org.apache.slider.common.SliderKeys;
+import org.apache.slider.common.params.AbstractActionArgs;
import org.apache.slider.common.params.AbstractClusterBuildingActionArgs;
import org.apache.slider.common.params.ActionAMSuicideArgs;
import org.apache.slider.common.params.ActionCreateArgs;
@@ -291,6 +292,12 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
// choose the action
String action = serviceArgs.getAction();
+
+AbstractActionArgs coreAction = serviceArgs.getCoreAction();
+if (coreAction.getHadoopServicesRequired()) {
+ // validate the client
+ SliderUtils.validateSliderClientEnvironment(null);
+}
int exitCode = EXIT_SUCCESS;
String clusterName = serviceArgs.getClusterName();
// actions
@@ -613,7 +620,9 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
// verify that a live cluster isn't there
SliderUtils.validateClusterName(clustername);
verifyBindingsDefined();
-if (!liveClusterAllowed) verifyNoLiveClusters(clustername);
+if (!liveClusterAllowed) {
+ verifyNoLiveClusters(clustername);
+}
Configuration conf = getConfig();
String registryQuorum = lookupZKQuorum();
@@ -1169,7 +1178,7 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
*/
addConfOptionToCLI(commandLine,
config,
- DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
+ DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY);
}
// write out the path output
commandLine.addOutAndErrFiles(STDOUT_AM, STDERR_AM);
@@ -1263,10 +1272,11 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
*/
private void propagatePrincipals(Configuration config,
AggregateConf clusterSpec) {
-String dfsPrincipal = config.get(DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
+String dfsPrincipal = config.get(
+DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY);
if (dfsPrincipal != null) {
String siteDfsPrincipal = OptionKeys.SITE_XML_PREFIX +
-DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY;
+
DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY;
clusterSpec.getAppConfOperations().getGlobalOptions().putIfUnset(
siteDfsPrincipal,
dfsPrincipal);
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b6361803/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
b/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
index f4a4569..d18c85b 100644
---
[4/6] git commit: SLIDER-377 AM checks for validity of server side; client does this if the CLI action says it should (default is yes)
SLIDER-377 AM checks for validity of server side; client does this if the CLI
action says it should (default is yes)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/01642770
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/01642770
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/01642770
Branch: refs/heads/develop
Commit: 0164277025ce3f8ec0a53b4dbd1005967632262e
Parents: b636180
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:04:46 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:04:46 2014 +0100
--
.../apache/slider/providers/agent/AgentTestBase.groovy| 10 ++
.../minicluster/masterless/TestRoleOptPropagation.groovy | 4 +---
2 files changed, 11 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/01642770/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
index 6dee64f..2953381 100644
---
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
@@ -26,9 +26,11 @@ import org.apache.commons.compress.utils.IOUtils
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.apache.slider.client.SliderClient
import org.apache.slider.common.params.SliderActions
+import org.apache.slider.common.tools.SliderUtils
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.test.YarnZKMiniClusterTestBase
import org.junit.Before
+import org.junit.BeforeClass
import org.junit.Rule
import org.junit.rules.TemporaryFolder
@@ -47,6 +49,14 @@ public abstract class AgentTestBase extends
YarnZKMiniClusterTestBase {
@Rule
public TemporaryFolder folder = new TemporaryFolder();
+ /**
+ * Server side test: validate system env before launch
+ */
+ @BeforeClass
+ public static void checkSystem() {
+SliderUtils.validateSliderServerEnvironment(null)
+ }
+
public String app_def_pkg_path;
@Before
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/01642770/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
--
diff --git
a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
index cffde39..dd9b8e9 100644
---
a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
+++
b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
@@ -30,8 +30,6 @@ import
org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
import org.apache.slider.core.main.ServiceLauncher
import org.junit.Test
-import static HBaseKeys.PROVIDER_HBASE
-import static Arguments.ARG_PROVIDER
@CompileStatic
@Slf4j
@@ -54,7 +52,7 @@ class TestRoleOptPropagation extends HBaseMiniClusterTestBase
{
],
[
Arguments.ARG_COMP_OPT, HBaseKeys.ROLE_MASTER, ENV, 4,
- ARG_PROVIDER, PROVIDER_HBASE
+ Arguments.ARG_PROVIDER, HBaseKeys.PROVIDER_HBASE
],
true,
true,
Git Push Summary
Repository: incubator-slider Updated Branches: refs/heads/feature/SLIDER-350-windows [deleted] 877a01609
Git Push Summary
Repository: incubator-slider Updated Branches: refs/heads/hadoop2.5/SLIDER-34_AM_Restart_not_working [deleted] cdbed7d7e
[1/6] git commit: SLIDER-394 slider client and AM services to fail-fast if the hadoop dependencies are missing
Repository: incubator-slider
Updated Branches:
refs/heads/develop 6320f6186 - 88d72ddfb
SLIDER-394 slider client and AM services to fail-fast if the hadoop
dependencies are missing
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0f67b104
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0f67b104
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0f67b104
Branch: refs/heads/develop
Commit: 0f67b10424cb807aebb3d2b15c3b24149f4ae70a
Parents: 29a40f5
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:28:20 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:28:20 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 555 +--
.../services/utility/EndOfServiceWaiter.java| 84 +++
.../services/workflow/ForkedProcessService.java | 21 +-
.../services/workflow/LongLivedProcess.java | 38 +-
.../tools/TestExecutionEnvironment.groovy | 42 ++
.../common/tools/TestWindowsSupport.groovy | 4 +-
.../services/workflow/EndOfServiceWaiter.java | 56 --
.../TestWorkflowForkedProcessService.java | 4 +-
8 files changed, 570 insertions(+), 234 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0f67b104/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index eb214db..44fe4fd 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -56,7 +56,10 @@ import org.apache.slider.core.exceptions.ErrorStrings;
import org.apache.slider.core.exceptions.MissingArgException;
import org.apache.slider.core.exceptions.SliderException;
import org.apache.slider.core.launch.ClasspathConstructor;
+import org.apache.slider.core.main.LauncherExitCodes;
+import org.apache.slider.server.services.utility.EndOfServiceWaiter;
import org.apache.slider.server.services.utility.PatternValidator;
+import org.apache.slider.server.services.workflow.ForkedProcessService;
import org.apache.zookeeper.server.util.KerberosUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -67,6 +70,7 @@ import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
+import java.io.InterruptedIOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.net.InetSocketAddress;
@@ -88,6 +92,7 @@ import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
import java.util.TreeSet;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
/**
@@ -102,12 +107,24 @@ public final class SliderUtils {
* turned on (prevents re-entrancy)
*/
private static final AtomicBoolean processSecurityAlreadyInitialized =
-new AtomicBoolean(false);
+ new AtomicBoolean(false);
public static final String JAVA_SECURITY_KRB5_REALM =
java.security.krb5.realm;
public static final String JAVA_SECURITY_KRB5_KDC = java.security.krb5.kdc;
+
+ /**
+ * Winutils
+ */
public static final String WINUTILS = WINUTILS.EXE;
+ /**
+ * name of openssl program
+ */
+ public static final String OPENSSL = openssl;
+ /**
+ * name of python program
+ */
+ public static final String PYTHON = python;
private SliderUtils() {
}
@@ -131,7 +148,8 @@ public final class SliderUtils {
* @param msg the message to be shown in exception
*/
@SuppressWarnings(ResultOfMethodCallIgnored)
- private static void validateNumber(String num, String msg) throws
BadConfigException {
+ private static void validateNumber(String num, String msg) throws
+ BadConfigException {
try {
Integer.parseInt(num);
} catch (NumberFormatException nfe) {
@@ -145,15 +163,16 @@ public final class SliderUtils {
* @param heapsize
* @return heapsize in MB
*/
- public static String translateTrailingHeapUnit(String heapsize) throws
BadConfigException {
+ public static String translateTrailingHeapUnit(String heapsize) throws
+ BadConfigException {
String errMsg = Bad heapsize: ;
if (heapsize.endsWith(m) || heapsize.endsWith(M)) {
- String num = heapsize.substring(0, heapsize.length()-1);
+ String num = heapsize.substring(0, heapsize.length() - 1);
validateNumber(num, errMsg);
return num;
}
if (heapsize.endsWith(g) || heapsize.endsWith(G)) {
- String num = heapsize.substring(0,
[5/6] git commit: SLIDER-377 AM-spawning tests do prelaunch validity checks
SLIDER-377 AM-spawning tests do prelaunch validity checks
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2d6f276d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2d6f276d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2d6f276d
Branch: refs/heads/develop
Commit: 2d6f276d16cc93648ccf7306eb94f5ac09bb5763
Parents: 0164277
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:09:12 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:09:12 2014 +0100
--
.../org/apache/slider/agent/AgentMiniClusterTestBase.groovy | 9 +
1 file changed, 9 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2d6f276d/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
index 4566d9d..7786c41 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
@@ -27,6 +27,7 @@ import org.apache.commons.io.FileUtils
import org.apache.slider.client.SliderClient
import org.apache.slider.common.SliderXMLConfKeysForTesting
import org.apache.slider.common.params.Arguments
+import org.apache.slider.common.tools.SliderUtils
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.providers.agent.AgentKeys
import org.apache.slider.test.YarnZKMiniClusterTestBase
@@ -47,6 +48,14 @@ extends YarnZKMiniClusterTestBase {
protected static MapString, String agentDefOptions
private static TemporaryFolder tempFolder = new TemporaryFolder();
+ /**
+ * Server side test: validate system env before launch
+ */
+ @BeforeClass
+ public static void checkSystem() {
+SliderUtils.validateSliderServerEnvironment(null)
+ }
+
@BeforeClass
public static void createSubConfFiles() {
[2/6] git commit: SLIDER-394 slider client and AM services to fail-fast if the hadoop dependencies are missing
SLIDER-394 slider client and AM services to fail-fast if the hadoop
dependencies are missing
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8034012f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8034012f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8034012f
Branch: refs/heads/develop
Commit: 8034012f2f0282c9dbab814a282ab47abc175204
Parents: 0f67b10
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:37:05 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:37:05 2014 +0100
--
.../slider/server/appmaster/SliderAppMaster.java| 3 +++
.../server/services/workflow/LongLivedProcess.java | 16 +++-
2 files changed, 18 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8034012f/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
index 9ccb382..49c7792 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
@@ -400,6 +400,9 @@ public class SliderAppMaster extends
AbstractSliderLaunchedService
//look at settings of Hadoop Auth, to pick up a problem seen once
checkAndWarnForAuthTokenProblems();
+
+// validate server env
+SliderUtils.validateSliderServerEnvironment(log);
executorService = new
WorkflowExecutorServiceExecutorService(AmExecutor,
Executors.newCachedThreadPool(
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8034012f/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
b/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
index 6f83811..a0df2c9 100644
---
a/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
+++
b/slider-core/src/main/java/org/apache/slider/server/services/workflow/LongLivedProcess.java
@@ -24,6 +24,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.BufferedReader;
+import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
@@ -292,7 +293,20 @@ public class LongLivedProcess implements Runnable {
if (LOG.isDebugEnabled()) {
LOG.debug(Spawning process:\n + describeBuilder());
}
-process = processBuilder.start();
+try {
+ process = processBuilder.start();
+} catch (IOException e) {
+ // on windows, upconvert DOS error 2 from ::CreateProcess()
+ // to its real meaning: FileNotFound
+ if (e.toString().contains(CreateProcess error=2)) {
+FileNotFoundException fnfe =
+new FileNotFoundException(e.toString());
+fnfe.initCause(e);
+throw fnfe;
+ } else {
+throw e;
+ }
+}
return process;
}
[2/3] git commit: SLIDER-377 tuning of some tests making a couple of methods accessible (groovy was apparently bending the rules)
SLIDER-377 tuning of some tests making a couple of methods accessible (groovy
was apparently bending the rules)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/4fc0396e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/4fc0396e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/4fc0396e
Branch: refs/heads/develop
Commit: 4fc0396e8705b0e50bbc6bd9671c1b8b5dbb3a0a
Parents: f2965d8
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 18:19:01 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 18:19:01 2014 +0100
--
.../org/apache/slider/client/SliderClient.java | 6 ++-
.../agent/actions/TestActionVersion.groovy | 5 +--
.../common/tools/TestZKIntegration.groovy | 44 ++--
.../slider/providers/agent/AgentTestBase.groovy | 2 +-
.../slider/test/YarnMiniClusterTestBase.groovy | 27 ++--
5 files changed, 43 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4fc0396e/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index f4040c4..ef6448d 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -355,7 +355,8 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
/**
* Delete the zookeeper node associated with the calling user and the cluster
**/
- protected boolean deleteZookeeperNode(String clusterName) throws
YarnException, IOException {
+ @VisibleForTesting
+ public boolean deleteZookeeperNode(String clusterName) throws YarnException,
IOException {
String user = getUsername();
String zkPath = ZKIntegration.mkClusterPath(user, clusterName);
Exception e = null;
@@ -391,7 +392,8 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
/**
* Create the zookeeper node associated with the calling user and the cluster
*/
- protected String createZookeeperNode(String clusterName, Boolean nameOnly)
throws YarnException, IOException {
+ @VisibleForTesting
+ public String createZookeeperNode(String clusterName, Boolean nameOnly)
throws YarnException, IOException {
String user = getUsername();
String zkPath = ZKIntegration.mkClusterPath(user, clusterName);
if(nameOnly) {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4fc0396e/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
index 1d50b71..7a2820f 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
@@ -20,10 +20,9 @@ package org.apache.slider.agent.actions
import groovy.util.logging.Slf4j
import org.apache.slider.common.params.SliderActions
-import org.apache.slider.test.YarnMiniClusterTestBase
+import org.apache.slider.test.SliderTestBase
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.apache.slider.core.main.ServiceLauncher
-import org.junit.Before
import org.junit.Test
/**
@@ -31,7 +30,7 @@ import org.junit.Test
*/
@Slf4j
-class TestActionVersion extends YarnMiniClusterTestBase {
+class TestActionVersion extends SliderTestBase {
@Test
public void testVersion() throws Throwable {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4fc0396e/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
index fe3bef7..c13af78 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
@@ -18,6 +18,7 @@
package org.apache.slider.common.tools
+import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
import org.apache.slider.client.SliderClient
@@ -31,12 +32,16 @@ import
[3/3] git commit: Merge branch 'feature/feature/SLIDER-377' into develop
Merge branch 'feature/feature/SLIDER-377' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a0a3ae63 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a0a3ae63 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a0a3ae63 Branch: refs/heads/develop Commit: a0a3ae632fc34be0137ac71b1564ca319c757632 Parents: 88d72dd 4fc0396 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 18:19:08 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 18:19:08 2014 +0100 -- .../org/apache/slider/client/SliderClient.java | 6 ++- .../agent/actions/TestActionVersion.groovy | 5 +-- .../tools/TestClientResourceRegistration.groovy | 3 ++ .../common/tools/TestConfigHelperHDFS.groovy| 3 ++ .../common/tools/TestMiscSliderUtils.groovy | 3 ++ .../common/tools/TestZKIntegration.groovy | 44 ++-- .../slider/providers/agent/AgentTestBase.groovy | 2 +- .../slider/test/YarnMiniClusterTestBase.groovy | 27 ++-- 8 files changed, 52 insertions(+), 41 deletions(-) --
git commit: SLIDER-393 fix pom to publish jars to ASF staging and snapshots (Giridharan Kesavan via billie)
Repository: incubator-slider
Updated Branches:
refs/heads/develop a0a3ae632 - 6126db62b
SLIDER-393 fix pom to publish jars to ASF staging and snapshots (Giridharan
Kesavan via billie)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6126db62
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6126db62
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6126db62
Branch: refs/heads/develop
Commit: 6126db62bb5d4ae45b6ecddb7b71ce1b8ae13cb2
Parents: a0a3ae6
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Thu Sep 4 10:40:03 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Thu Sep 4 10:40:03 2014 -0700
--
pom.xml | 41 +++--
1 file changed, 39 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6126db62/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8292073..9efb5c8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -72,6 +72,19 @@
urlhttp://slider.incubator.apache.org//url
/site
downloadUrlhttp://git-wip-us.apache.org/repos/asf/incubator-slider.git/downloadUrl
+repository
+ id${distMgmtStagingId}/id
+ name${distMgmtStagingName}/name
+ url${distMgmtStagingUrl}/url
+/repository
+snapshotRepository
+ id${distMgmtSnapshotsId}/id
+ name${distMgmtSnapshotsName}/name
+ url${distMgmtSnapshotsUrl}/url
+/snapshotRepository
+
+
+
/distributionManagement
mailingLists
@@ -92,7 +105,12 @@
/mailingLists
properties
-
+distMgmtSnapshotsIdapache.snapshots.https/distMgmtSnapshotsId
+distMgmtSnapshotsNameApache Development Snapshot
Repository/distMgmtSnapshotsName
+
distMgmtSnapshotsUrlhttps://repository.apache.org/content/repositories/snapshots/distMgmtSnapshotsUrl
+distMgmtStagingIdapache.staging.https/distMgmtStagingId
+distMgmtStagingNameApache Release Distribution
Repository/distMgmtStagingName
+
distMgmtStagingUrlhttps://repository.apache.org/service/local/staging/deploy/maven2/distMgmtStagingUrl
project.build.sourceEncodingUTF-8/project.build.sourceEncoding
project.reporting.outputEncodingUTF-8/project.reporting.outputEncoding
@@ -1397,7 +1415,26 @@
/build
/profile
-
+profile
+ idsign/id
+ build
+plugins
+ plugin
+groupIdorg.apache.maven.plugins/groupId
+artifactIdmaven-gpg-plugin/artifactId
+executions
+ execution
+idsign-artifacts/id
+phaseverify/phase
+goals
+ goalsign/goal
+/goals
+ /execution
+/executions
+ /plugin
+/plugins
+ /build
+/profile
/profiles
[3/3] git commit: Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop
Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/930920d2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/930920d2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/930920d2 Branch: refs/heads/develop Commit: 930920d245bb354cbfbec43391a203bd6db04fa4 Parents: b7f37ea 6126db6 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 19:00:55 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 19:00:55 2014 +0100 -- pom.xml | 41 +++-- 1 file changed, 39 insertions(+), 2 deletions(-) --
[2/3] git commit: SLIDER-395 testZKIntegration broken
SLIDER-395 testZKIntegration broken
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b7f37ea2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b7f37ea2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b7f37ea2
Branch: refs/heads/develop
Commit: b7f37ea2b1f6842fab57b37a6b54b08209b895aa
Parents: bd0cdd1
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 19:00:42 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 19:00:42 2014 +0100
--
.../slider/common/tools/TestZKIntegration.groovy| 16 +++-
.../slider/test/YarnZKMiniClusterTestBase.groovy| 2 +-
2 files changed, 12 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b7f37ea2/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
index c13af78..836edbd 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
@@ -48,7 +48,8 @@ class TestZKIntegration extends YarnZKMiniClusterTestBase
implements KeysForTest
@Test
public void testIntegrationCreate() throws Throwable {
assertHasZKCluster()
-ZKIntegration zki = createZKIntegrationInstance(ZKBinding, cluster1,
true, false, 5000)
+ZKIntegration zki = createZKIntegrationInstance(
+getZKBinding(), cluster1, true, false, 5000)
String userPath = ZKIntegration.mkSliderUserPath(USER)
Stat stat = zki.stat(userPath)
assert stat != null
@@ -59,7 +60,8 @@ class TestZKIntegration extends YarnZKMiniClusterTestBase
implements KeysForTest
public void testListUserClustersWithoutAnyClusters() throws Throwable {
assertHasZKCluster()
-ZKIntegration zki = createZKIntegrationInstance(ZKBinding, , true,
false, 5000)
+ZKIntegration zki = createZKIntegrationInstance(
+getZKBinding(), , true, false, 5000)
String userPath = ZKIntegration.mkSliderUserPath(USER)
ListString clusters = zki.clusters
assert clusters.empty
@@ -69,7 +71,8 @@ class TestZKIntegration extends YarnZKMiniClusterTestBase
implements KeysForTest
public void testListUserClustersWithOneCluster() throws Throwable {
assertHasZKCluster()
-ZKIntegration zki = createZKIntegrationInstance(ZKBinding, , true,
false, 5000)
+ZKIntegration zki = createZKIntegrationInstance(
+getZKBinding(), , true, false, 5000)
String userPath = ZKIntegration.mkSliderUserPath(USER)
String fullPath = zki.createPath(userPath, /cluster-,
ZooDefs.Ids.OPEN_ACL_UNSAFE,
@@ -82,7 +85,8 @@ class TestZKIntegration extends YarnZKMiniClusterTestBase
implements KeysForTest
@Test
public void testListUserClustersWithTwoCluster() throws Throwable {
-ZKIntegration zki = createZKIntegrationInstance(ZKBinding, , true,
false, 5000)
+ZKIntegration zki = createZKIntegrationInstance(
+getZKBinding(), , true, false, 5000)
String userPath = ZKIntegration.mkSliderUserPath(USER)
String c1 = createEphemeralChild(zki, userPath)
log.info(Ephemeral path $c1)
@@ -110,6 +114,7 @@ class TestZKIntegration extends YarnZKMiniClusterTestBase
implements KeysForTest
path = client.createZookeeperNode(cl1, false)
zki = client.lastZKIntegration
+assert zki
assert zkPath == /services/slider/users/ + USER + /cl1, zkPath must
be as expected
assert path == zkPath
assert zki.exists(zkPath)
@@ -135,7 +140,8 @@ class TestZKIntegration extends YarnZKMiniClusterTestBase
implements KeysForTest
@Override
protected ZKIntegration getZkClient(String clusterName, String user) {
- createZKIntegrationInstance(ZKBinding, cl1, true, false, 5000)
+ zki = createZKIntegrationInstance(getZKBinding(), cl1, true, false,
5000)
+ return zki;
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b7f37ea2/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
index 0259fb7..9b5f082 100644
---
a/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
+++
git commit: SLIDER-397 windows kill() command failing in TestStandaloneAMKill
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-397_windows_test_kill()_command_failing [created]
132d9f288
SLIDER-397 windows kill() command failing in TestStandaloneAMKill
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/132d9f28
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/132d9f28
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/132d9f28
Branch: refs/heads/feature/SLIDER-397_windows_test_kill()_command_failing
Commit: 132d9f288a9d91790409bd4482731eb27cf58bba
Parents: 930920d
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 20:31:29 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 20:31:29 2014 +0100
--
.../apache/slider/agent/standalone/TestStandaloneAMKill.groovy | 2 +-
.../groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/132d9f28/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
index 75f9a2c..7552394 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
@@ -58,7 +58,7 @@ class TestStandaloneAMKill extends AgentMiniClusterTestBase {
describe(listing Java processes)
lsJavaProcesses();
describe(killing AM)
-killAM(SIGTERM);
+assert 0 == killAM(SIGTERM);
waitWhileClusterLive(sliderClient);
//give yarn some time to notice
sleep(1)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/132d9f28/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
index 7ffa64a..97cc853 100644
---
a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
@@ -318,10 +318,10 @@ public abstract class YarnMiniClusterTestBase extends
ServiceLauncherBaseTest {
/**
- * Kill all Slider Services. That i
+ * Kill all Slider Services.
* @param signal
*/
- public void killAM(int signal) {
+ public int killAM(int signal) {
killJavaProcesses(SliderAppMaster.SERVICE_CLASSNAME_SHORT, signal)
}
[22/50] git commit: SLIDER-388 allow default zookeeper port in ZookeeperUtils
SLIDER-388 allow default zookeeper port in ZookeeperUtils
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b525287d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b525287d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b525287d
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: b525287dd05a29af6308890aef44d524a6341296
Parents: 0ac2536
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Tue Sep 2 12:39:06 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Tue Sep 2 12:39:06 2014 -0700
--
.../src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b525287d/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
b/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
index 61b1ff0..cc1b2c9 100644
--- a/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
@@ -27,6 +27,7 @@ import java.util.ArrayList;
import java.util.List;
public class ZookeeperUtils {
+ public static final int DEFAULT_PORT = 2181;
public static String buildConnectionString(String zkHosts, int port) {
String zkPort = Integer.toString(port);
@@ -73,7 +74,7 @@ public class ZookeeperUtils {
ListHostAndPort list = new ArrayListHostAndPort(len);
if (strings != null) {
for (String s : strings) {
-list.add(HostAndPort.fromString(s.trim()));
+
list.add(HostAndPort.fromString(s.trim()).withDefaultPort(DEFAULT_PORT));
}
}
return list;
[03/50] git commit: SLIDER-371 test teardown warning of deletion failures ... fix root cause in SliderUtils
SLIDER-371 test teardown warning of deletion failures ... fix root cause in
SliderUtils
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/4d5783e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/4d5783e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/4d5783e1
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 4d5783e12e3f2a43bfe02e15bf789d34341af0f0
Parents: e4d136e
Author: Steve Loughran ste...@apache.org
Authored: Thu Aug 28 16:45:28 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Aug 28 18:28:46 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 55 +++-
.../agent/AgentMiniClusterTestBase.groovy | 2 +-
2 files changed, 31 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4d5783e1/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index 188b7d9..a864878 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -1460,35 +1460,40 @@ public final class SliderUtils {
String entry)
throws IOException {
InputStream is = null;
-FSDataInputStream appStream = fs.open(appPath);
-ZipArchiveInputStream zis = new ZipArchiveInputStream(appStream);
-ZipArchiveEntry zipEntry;
-boolean done = false;
-while (!done (zipEntry = zis.getNextZipEntry()) != null) {
- if (entry.equals(zipEntry.getName())) {
-int size = (int) zipEntry.getSize();
-if (size != -1) {
- log.info(Reading {} of size {}, zipEntry.getName(),
zipEntry.getSize());
- byte[] content = new byte[size];
- int offset = 0;
- while (offset size) {
-offset += zis.read(content, offset, size - offset);
- }
- is = new ByteArrayInputStream(content);
-} else {
- log.debug(Size unknown. Reading {}, zipEntry.getName());
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- while (true) {
-int byteRead = zis.read();
-if (byteRead == -1) {
- break;
+FSDataInputStream appStream = null;
+try {
+ appStream = fs.open(appPath);
+ ZipArchiveInputStream zis = new ZipArchiveInputStream(appStream);
+ ZipArchiveEntry zipEntry;
+ boolean done = false;
+ while (!done (zipEntry = zis.getNextZipEntry()) != null) {
+if (entry.equals(zipEntry.getName())) {
+ int size = (int) zipEntry.getSize();
+ if (size != -1) {
+log.info(Reading {} of size {}, zipEntry.getName(),
zipEntry.getSize());
+byte[] content = new byte[size];
+int offset = 0;
+while (offset size) {
+ offset += zis.read(content, offset, size - offset);
}
-baos.write(byteRead);
+is = new ByteArrayInputStream(content);
+ } else {
+log.debug(Size unknown. Reading {}, zipEntry.getName());
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+while (true) {
+ int byteRead = zis.read();
+ if (byteRead == -1) {
+break;
+ }
+ baos.write(byteRead);
+}
+is = new ByteArrayInputStream(baos.toByteArray());
}
- is = new ByteArrayInputStream(baos.toByteArray());
+ done = true;
}
-done = true;
}
+} finally {
+ IOUtils.closeStream(appStream);
}
return is;
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4d5783e1/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
index a957481..4566d9d 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
@@ -85,7 +85,7 @@ extends YarnZKMiniClusterTestBase {
try {
FileUtils.deleteDirectory(tempRoot);
} catch (IOException e) {
-log.warn(Failed to delete $tempRoot :$e, e)
+
[49/50] git commit: Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop
Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/930920d2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/930920d2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/930920d2 Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 930920d245bb354cbfbec43391a203bd6db04fa4 Parents: b7f37ea 6126db6 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 19:00:55 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 19:00:55 2014 +0100 -- pom.xml | 41 +++-- 1 file changed, 39 insertions(+), 2 deletions(-) --
[04/50] git commit: SLIDER-366. App package clean up - remove extra information from appConfig.json for memcached.
SLIDER-366. App package clean up - remove extra information from appConfig.json
for memcached.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/f292a16e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/f292a16e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/f292a16e
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: f292a16ebfed4d75711f87577f4cde63698e3704
Parents: 4d5783e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Aug 28 12:51:49 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Aug 28 12:51:49 2014 -0700
--
app-packages/memcached/appConfig.json | 2 --
1 file changed, 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f292a16e/app-packages/memcached/appConfig.json
--
diff --git a/app-packages/memcached/appConfig.json
b/app-packages/memcached/appConfig.json
index 7a97f98..c335959 100644
--- a/app-packages/memcached/appConfig.json
+++ b/app-packages/memcached/appConfig.json
@@ -19,8 +19,6 @@
components: {
slider-appmaster: {
jvm.heapsize: 256M
-},
-MEMCACHED: {
}
}
}
[05/50] git commit: SLIDER-258. Jenkins build continue to hang on python unit tests
SLIDER-258. Jenkins build continue to hang on python unit tests
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8bfd66e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8bfd66e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8bfd66e8
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 8bfd66e813635aa93a9551c121622b408d41b518
Parents: f292a16
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Aug 28 15:37:31 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Aug 28 15:37:31 2014 -0700
--
.../src/test/python/agent/TestRegistration.py | 14 +-
slider-agent/src/test/python/unitTests.py | 1 +
2 files changed, 2 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8bfd66e8/slider-agent/src/test/python/agent/TestRegistration.py
--
diff --git a/slider-agent/src/test/python/agent/TestRegistration.py
b/slider-agent/src/test/python/agent/TestRegistration.py
index f91fe29..7b3c875 100644
--- a/slider-agent/src/test/python/agent/TestRegistration.py
+++ b/slider-agent/src/test/python/agent/TestRegistration.py
@@ -37,15 +37,6 @@ class TestRegistration(TestCase):
config = AgentConfig(tmpdir, ver_dir)
config.set('agent', 'prefix', tmpdir)
config.set('agent', 'current_ping_port', '33777')
-try:
- os.mkdir(ver_dir)
-except OSError as exception:
- if exception.errno != errno.EEXIST:
-raise
-pass
-ver_file = os.path.join(ver_dir, version)
-with open(ver_file, w) as text_file:
- text_file.write(1.3.0)
register = Register(config)
data = register.build(State.INIT, State.INIT, {}, 1)
@@ -54,7 +45,7 @@ class TestRegistration(TestCase):
self.assertEquals(data['publicHostname'] != , True, publicHostname
should not be empty)
self.assertEquals(data['responseId'], 1)
self.assertEquals(data['timestamp'] 1353678475465L, True, timestamp
should not be empty)
-self.assertEquals(data['agentVersion'], '1.3.0', agentVersion should not
be empty)
+self.assertEquals(data['agentVersion'], '1', agentVersion should not be
empty)
self.assertEquals(data['actualState'], State.INIT, actualState should not
be empty)
self.assertEquals(data['expectedState'], State.INIT, expectedState should
not be empty)
self.assertEquals(data['allocatedPorts'], {}, allocatedPorts should be
empty)
@@ -66,8 +57,5 @@ class TestRegistration(TestCase):
self.assertEquals(os.path.join(ver_dir, .),
config.getResolvedPath(log_dir))
self.assertEquals(os.path.join(ver_dir, .),
config.getResolvedPath(app_task_dir))
-os.remove(ver_file)
-os.removedirs(ver_dir)
-
if __name__ == __main__:
unittest.main()
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8bfd66e8/slider-agent/src/test/python/unitTests.py
--
diff --git a/slider-agent/src/test/python/unitTests.py
b/slider-agent/src/test/python/unitTests.py
index e3f2d7c..b01438e 100644
--- a/slider-agent/src/test/python/unitTests.py
+++ b/slider-agent/src/test/python/unitTests.py
@@ -56,6 +56,7 @@ def all_tests_suite():
files_list += os.listdir(src_dir + os.sep + directory)
## temporarily deleting to add more predictability
## shuffle(files_list)
+ files_list.sort()
tests_list = []
logger.info('TESTS
LIST:-')
[15/50] git commit: SLIDER-379. Modify default appConfig.json for he HBase package
SLIDER-379. Modify default appConfig.json for he HBase package
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/ffbc3a3c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/ffbc3a3c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/ffbc3a3c
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: ffbc3a3c31d762d8635ab11394a608c74b080582
Parents: 11f6f2b
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sat Aug 30 07:42:06 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sat Aug 30 07:42:06 2014 -0700
--
app-packages/hbase/appConfig.json | 76 +-
1 file changed, 38 insertions(+), 38 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/ffbc3a3c/app-packages/hbase/appConfig.json
--
diff --git a/app-packages/hbase/appConfig.json
b/app-packages/hbase/appConfig.json
index d324e6d..7df0dc3 100644
--- a/app-packages/hbase/appConfig.json
+++ b/app-packages/hbase/appConfig.json
@@ -1,45 +1,45 @@
{
- schema: http://example.org/specification/v2.0.0;,
- metadata: {
- },
- global: {
-application.def: ${app.package.name}.zip,
-create.default.zookeeper.node: true,
-java_home: /usr/jdk64/jdk1.7.0_45,
-system_configs: core-site,
+schema: http://example.org/specification/v2.0.0;,
+metadata: {
+},
+global: {
+application.def: ${app.package.name}.zip,
+create.default.zookeeper.node: true,
+java_home: /usr/jdk64/jdk1.7.0_45,
+system_configs: core-site,
-site.global.app_user: yarn,
-site.global.app_root:
${AGENT_WORK_ROOT}/app/install/hbase-${hbase.version},
+site.global.app_user: yarn,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/hbase-${hbase.version},
-site.global.hbase_instance_name: instancename,
-site.global.hbase_root_password: secret,
-site.global.user_group: hadoop,
-site.global.security_enabled: false,
-site.global.monitor_protocol: http,
-site.global.ganglia_server_host: ${NN_HOST},
-site.global.ganglia_server_port: 8667,
-site.global.ganglia_server_id: Application1,
-site.global.hbase_thrift_port: ${HBASE_THRIFT.ALLOCATED_PORT},
-site.global.hbase_thrift2_port: ${HBASE_THRIFT2.ALLOCATED_PORT},
-site.global.hbase_rest_port: ${HBASE_REST.ALLOCATED_PORT},
+site.global.hbase_instance_name: instancename,
+site.global.hbase_root_password: secret,
+site.global.user_group: hadoop,
+site.global.security_enabled: false,
+site.global.monitor_protocol: http,
+site.global.ganglia_server_host: ${NN_HOST},
+site.global.ganglia_server_port: 8667,
+site.global.ganglia_server_id: Application1,
+site.global.hbase_thrift_port: ${HBASE_THRIFT.ALLOCATED_PORT},
+site.global.hbase_thrift2_port: ${HBASE_THRIFT2.ALLOCATED_PORT},
+site.global.hbase_rest_port: ${HBASE_REST.ALLOCATED_PORT},
-site.hbase-env.hbase_master_heapsize: 1024m,
-site.hbase-env.hbase_regionserver_heapsize: 1024m,
+site.hbase-env.hbase_master_heapsize: 1024m,
+site.hbase-env.hbase_regionserver_heapsize: 1024m,
-site.hbase-site.hbase.rootdir: ${DEFAULT_DATA_DIR},
-site.hbase-site.hbase.superuser: yarn,
-site.hbase-site.hbase.tmp.dir: ${AGENT_WORK_ROOT}/work/app/tmp,
-site.hbase-site.hbase.local.dir: ${hbase.tmp.dir}/local,
-site.hbase-site.hbase.zookeeper.quorum: ${ZK_HOST},
-site.hbase-site.zookeeper.znode.parent: ${DEF_ZK_PATH},
-site.hbase-site.hbase.regionserver.info.port: 0,
-site.hbase-site.hbase.master.info.port: ${HBASE_MASTER.ALLOCATED_PORT},
- site.hbase-site.hbase.regionserver.port: 0,
- site.hbase-site.hbase.master.port: 0
- },
- components: {
-slider-appmaster: {
- jvm.heapsize: 256M
+site.hbase-site.hbase.rootdir: ${DEFAULT_DATA_DIR},
+site.hbase-site.hbase.superuser: yarn,
+site.hbase-site.hbase.tmp.dir: ${AGENT_WORK_ROOT}/work/app/tmp,
+site.hbase-site.hbase.local.dir: ${hbase.tmp.dir}/local,
+site.hbase-site.hbase.zookeeper.quorum: ${ZK_HOST},
+site.hbase-site.zookeeper.znode.parent: ${DEF_ZK_PATH},
+site.hbase-site.hbase.regionserver.info.port: 0,
+site.hbase-site.hbase.master.info.port:
${HBASE_MASTER.ALLOCATED_PORT},
+site.hbase-site.hbase.regionserver.port: 0,
+site.hbase-site.hbase.master.port: 0
+},
+components: {
+slider-appmaster: {
+jvm.heapsize: 256M
+}
}
- }
}
[43/50] git commit: SLIDER-377 tagging some test files as @CompileStatic
SLIDER-377 tagging some test files as @CompileStatic
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/f2965d8a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/f2965d8a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/f2965d8a
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: f2965d8a056fcf6d56a39f0841f1903048c95c29
Parents: 88d72dd
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:54:46 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:54:46 2014 +0100
--
.../slider/common/tools/TestClientResourceRegistration.groovy | 3 +++
.../org/apache/slider/common/tools/TestConfigHelperHDFS.groovy| 3 +++
.../org/apache/slider/common/tools/TestMiscSliderUtils.groovy | 3 +++
3 files changed, 9 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f2965d8a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestClientResourceRegistration.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestClientResourceRegistration.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestClientResourceRegistration.groovy
index 7150b3c..e3bfa1b 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestClientResourceRegistration.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestClientResourceRegistration.groovy
@@ -18,12 +18,15 @@
package org.apache.slider.common.tools
+import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.junit.Test
@Slf4j
+@CompileStatic
+
class TestClientResourceRegistration {
/**
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f2965d8a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestConfigHelperHDFS.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestConfigHelperHDFS.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestConfigHelperHDFS.groovy
index 0d21d6c..d78aa1b 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestConfigHelperHDFS.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestConfigHelperHDFS.groovy
@@ -18,6 +18,7 @@
package org.apache.slider.common.tools
+import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem as HadoopFS
@@ -27,6 +28,8 @@ import org.apache.slider.test.YarnMiniClusterTestBase
import org.junit.Test
@Slf4j
+@CompileStatic
+
class TestConfigHelperHDFS extends YarnMiniClusterTestBase {
//diabled for now;
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f2965d8a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestMiscSliderUtils.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestMiscSliderUtils.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestMiscSliderUtils.groovy
index 24367a3..9042af5 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestMiscSliderUtils.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestMiscSliderUtils.groovy
@@ -18,12 +18,15 @@
package org.apache.slider.common.tools
+import groovy.transform.CompileStatic
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem as HadoopFS
import org.apache.hadoop.fs.Path
import org.apache.slider.test.SliderTestBase
import org.junit.Test
+@CompileStatic
+
class TestMiscSliderUtils extends SliderTestBase {
[33/50] git commit: SLIDER-350 windows support
SLIDER-350 windows support
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/93c3c770
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/93c3c770
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/93c3c770
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 93c3c770fed878895e4fb67ac17cbbb098cb4fb3
Parents: 8f9caa9
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:05:38 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:05:38 2014 +0100
--
.../common/tools/TestWindowsSupport.groovy | 22 +++-
1 file changed, 17 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/93c3c770/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index 61ede62..6ae683a 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -109,13 +109,13 @@ class TestWindowsSupport extends SliderTestBase {
@Test
public void testHasGawkInstalled() throws Throwable {
assume(Shell.WINDOWS, not windows)
-assert 0 == exec([gawk, --version])
+exec(0, [gawk, --version])
}
@Test
public void testHasXargsInstalled() throws Throwable {
assume(Shell.WINDOWS, not windows)
-assert 0 == exec([xargs, --version])
+exec(0, [xargs, --version])
}
@@ -145,16 +145,28 @@ class TestWindowsSupport extends SliderTestBase {
File winUtils = new File(winUtilsPath)
log.debug(Winutils is at $winUtils))
-ForkedProcessService process = exec([winUtilsPath, systeminfo])
-assert 0 == process.exitCode
+exec(0, [winUtilsPath, systeminfo])
}
/**
* Exec a set of commands, wait a few seconds for it to finish.
+ * @param status code
* @param commands
- * @return
+ * @return the process
*/
+ public ForkedProcessService exec(int status, ListString commands) {
+ForkedProcessService process = exec(commands)
+assert status == process.exitCode
+return process
+ }
+
+ /**
+ * Exec a set of commands, wait a few seconds for it to finish.
+ * @param commands
+ * @return
+ */
+
public ForkedProcessService exec(ListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
[02/50] git commit: SLIDER-368 change app package names to standard maven artifact names
SLIDER-368 change app package names to standard maven artifact names
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e4d136e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e4d136e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e4d136e1
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: e4d136e152161fa4c7a8f648773797d8b2541867
Parents: b336c57
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Thu Aug 28 08:36:17 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Thu Aug 28 08:36:17 2014 -0700
--
app-packages/accumulo/pom.xml | 2 +-
app-packages/hbase/pom.xml| 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e4d136e1/app-packages/accumulo/pom.xml
--
diff --git a/app-packages/accumulo/pom.xml b/app-packages/accumulo/pom.xml
index 40d6ba7..548ced8 100644
--- a/app-packages/accumulo/pom.xml
+++ b/app-packages/accumulo/pom.xml
@@ -30,7 +30,7 @@
properties
work.dirpackage-tmp/work.dir
-
app.package.nameapache-slider-accumulo-${accumulo.version}-app-package-${project.version}/app.package.name
+
app.package.name${project.artifactId}-${project.version}/app.package.name
/properties
profiles
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e4d136e1/app-packages/hbase/pom.xml
--
diff --git a/app-packages/hbase/pom.xml b/app-packages/hbase/pom.xml
index 7dede6c..442b0f2 100644
--- a/app-packages/hbase/pom.xml
+++ b/app-packages/hbase/pom.xml
@@ -29,7 +29,7 @@
descriptionSlider HBase App Package/description
properties
work.dirpackage-tmp/work.dir
-
app.package.nameapache-slider-hbase-${hbase.version}-app-package-${project.version}/app.package.name
+
app.package.name${project.artifactId}-${project.version}/app.package.name
/properties
profiles
[36/50] git commit: SLIDER-146 added sample appConfig that enables kerberos for Accumulo
SLIDER-146 added sample appConfig that enables kerberos for Accumulo
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6320f618
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6320f618
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6320f618
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 6320f6186dec56f68afbbd638f5ddc0e7dc04bff
Parents: 29a40f5
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Thu Sep 4 07:15:22 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Thu Sep 4 07:15:22 2014 -0700
--
.../src/test/resources/appConfig_kerberos.json | 55
1 file changed, 55 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6320f618/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
--
diff --git a/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
b/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
new file mode 100644
index 000..c9684d4
--- /dev/null
+++ b/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
@@ -0,0 +1,55 @@
+{
+ schema: http://example.org/specification/v2.0.0;,
+ metadata: {
+ },
+ global: {
+application.def: ${app.package.name}.zip,
+java_home: /usr/lib/jvm/java,
+site.global.app_user: accumulo,
+site.global.app_log_dir: ${AGENT_LOG_ROOT},
+site.global.app_pid_dir: ${AGENT_WORK_ROOT}/app/run,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/accumulo-${accumulo.version},
+site.global.tserver_heapsize: 256m,
+site.global.master_heapsize: 128m,
+site.global.monitor_heapsize: 64m,
+site.global.gc_heapsize: 64m,
+site.global.other_heapsize: 128m,
+site.global.hadoop_prefix: /usr/lib/hadoop,
+site.global.hadoop_conf_dir: /etc/hadoop/conf,
+site.global.zookeeper_home: /usr/lib/zookeeper,
+site.global.accumulo_instance_name: ${USER}-${CLUSTER_NAME},
+site.global.accumulo_root_password: NOT_USED,
+site.global.user_group: accumulo,
+site.global.security_enabled: false,
+site.global.ssl_cert_dir: ssl,
+site.global.monitor_protocol: http,
+site.accumulo-site.instance.volumes: ${DEFAULT_DATA_DIR}/data,
+site.accumulo-site.instance.zookeeper.host: ${ZK_HOST},
+site.accumulo-site.instance.security.authenticator:
org.apache.slider.accumulo.CustomAuthenticator,
+site.accumulo-site.general.security.credential.provider.paths:
jceks://hdfs/user/${USER}/accumulo-${CLUSTER_NAME}.jceks,
+site.accumulo-site.general.kerberos.keytab:
/etc/security/keytabs/accumulo.service.keytab,
+site.accumulo-site.general.kerberos.principal:
accumulo/_h...@example.com,
+site.accumulo-site.tserver.memory.maps.native.enabled: false,
+site.accumulo-site.tserver.memory.maps.max: 80M,
+site.accumulo-site.tserver.cache.data.size: 7M,
+site.accumulo-site.tserver.cache.index.size: 20M,
+site.accumulo-site.trace.user: root,
+site.accumulo-site.tserver.sort.buffer.size: 50M,
+site.accumulo-site.tserver.walog.max.size: 40M,
+site.accumulo-site.master.port.client: 0,
+site.accumulo-site.trace.port.client: 0,
+site.accumulo-site.tserver.port.client: 0,
+site.accumulo-site.gc.port.client: 0,
+site.accumulo-site.monitor.port.client:
${ACCUMULO_MONITOR.ALLOCATED_PORT},
+site.accumulo-site.monitor.port.log4j: 0,
+site.accumulo-site.general.classpaths:
$ACCUMULO_HOME/lib/accumulo-server.jar,\n$ACCUMULO_HOME/lib/accumulo-core.jar,\n$ACCUMULO_HOME/lib/accumulo-start.jar,\n$ACCUMULO_HOME/lib/accumulo-fate.jar,\n$ACCUMULO_HOME/lib/accumulo-proxy.jar,\n$ACCUMULO_HOME/lib/[^.].*.jar,\n$ZOOKEEPER_HOME/zookeeper[^.].*.jar,\n$HADOOP_CONF_DIR,\n$HADOOP_PREFIX/[^.].*.jar,\n$HADOOP_PREFIX/lib/[^.].*.jar,\n$HADOOP_PREFIX/share/hadoop/common/.*.jar,\n$HADOOP_PREFIX/share/hadoop/common/lib/.*.jar,\n$HADOOP_PREFIX/share/hadoop/hdfs/.*.jar,\n$HADOOP_PREFIX/share/hadoop/mapreduce/.*.jar,\n$HADOOP_PREFIX/share/hadoop/yarn/.*.jar,\n/usr/lib/hadoop/.*.jar,\n/usr/lib/hadoop/lib/.*.jar,\n/usr/lib/hadoop-hdfs/.*.jar,\n/usr/lib/hadoop-mapreduce/.*.jar,\n/usr/lib/hadoop-yarn/.*.jar,
+ },
+ credentials: {
+jceks://hdfs/user/${USER}/accumulo-${CLUSTER_NAME}.jceks:
[root.initial.password, instance.secret, trace.token.property.password]
+ },
+ components: {
+slider-appmaster: {
+ jvm.heapsize: 256M
+}
+ }
+}
[42/50] git commit: Merge branch 'feature/SLIDER-394_env_check' into develop
Merge branch 'feature/SLIDER-394_env_check' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/88d72ddf Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/88d72ddf Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/88d72ddf Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 88d72ddfb32977ff098b0329ae6f3faf2f7c2875 Parents: 6320f61 2d6f276 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 17:12:02 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 17:12:02 2014 +0100 -- .../org/apache/slider/client/SliderClient.java | 18 +- .../common/params/AbstractActionArgs.java | 11 + .../slider/common/params/ActionHelpArgs.java| 14 +- .../slider/common/params/ActionVersionArgs.java | 12 +- .../apache/slider/common/tools/SliderUtils.java | 557 +-- .../slider/core/build/InstanceBuilder.java | 5 +- .../server/appmaster/SliderAppMaster.java | 5 +- .../services/utility/EndOfServiceWaiter.java| 84 +++ .../services/workflow/ForkedProcessService.java | 21 +- .../services/workflow/LongLivedProcess.java | 57 +- .../agent/AgentMiniClusterTestBase.groovy | 9 + .../tools/TestExecutionEnvironment.groovy | 42 ++ .../common/tools/TestWindowsSupport.groovy | 4 +- .../slider/providers/agent/AgentTestBase.groovy | 10 + .../services/workflow/EndOfServiceWaiter.java | 56 -- .../TestWorkflowForkedProcessService.java | 4 +- .../masterless/TestRoleOptPropagation.groovy| 4 +- 17 files changed, 663 insertions(+), 250 deletions(-) --
[40/50] git commit: SLIDER-377 AM checks for validity of server side; client does this if the CLI action says it should (default is yes)
SLIDER-377 AM checks for validity of server side; client does this if the CLI
action says it should (default is yes)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/01642770
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/01642770
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/01642770
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 0164277025ce3f8ec0a53b4dbd1005967632262e
Parents: b636180
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:04:46 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:04:46 2014 +0100
--
.../apache/slider/providers/agent/AgentTestBase.groovy| 10 ++
.../minicluster/masterless/TestRoleOptPropagation.groovy | 4 +---
2 files changed, 11 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/01642770/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
index 6dee64f..2953381 100644
---
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/AgentTestBase.groovy
@@ -26,9 +26,11 @@ import org.apache.commons.compress.utils.IOUtils
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.apache.slider.client.SliderClient
import org.apache.slider.common.params.SliderActions
+import org.apache.slider.common.tools.SliderUtils
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.test.YarnZKMiniClusterTestBase
import org.junit.Before
+import org.junit.BeforeClass
import org.junit.Rule
import org.junit.rules.TemporaryFolder
@@ -47,6 +49,14 @@ public abstract class AgentTestBase extends
YarnZKMiniClusterTestBase {
@Rule
public TemporaryFolder folder = new TemporaryFolder();
+ /**
+ * Server side test: validate system env before launch
+ */
+ @BeforeClass
+ public static void checkSystem() {
+SliderUtils.validateSliderServerEnvironment(null)
+ }
+
public String app_def_pkg_path;
@Before
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/01642770/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
--
diff --git
a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
index cffde39..dd9b8e9 100644
---
a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
+++
b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/masterless/TestRoleOptPropagation.groovy
@@ -30,8 +30,6 @@ import
org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
import org.apache.slider.core.main.ServiceLauncher
import org.junit.Test
-import static HBaseKeys.PROVIDER_HBASE
-import static Arguments.ARG_PROVIDER
@CompileStatic
@Slf4j
@@ -54,7 +52,7 @@ class TestRoleOptPropagation extends HBaseMiniClusterTestBase
{
],
[
Arguments.ARG_COMP_OPT, HBaseKeys.ROLE_MASTER, ENV, 4,
- ARG_PROVIDER, PROVIDER_HBASE
+ Arguments.ARG_PROVIDER, HBaseKeys.PROVIDER_HBASE
],
true,
true,
[26/50] git commit: SLIDER-390 interpret short lived threshold as seconds instead of milliseconds
SLIDER-390 interpret short lived threshold as seconds instead of milliseconds
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b0b7bd38
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b0b7bd38
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b0b7bd38
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: b0b7bd38542ed287415066d0215c6fda8786336e
Parents: 201686e
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Wed Sep 3 08:38:52 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Wed Sep 3 08:38:52 2014 -0700
--
.../java/org/apache/slider/server/appmaster/state/AppState.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b0b7bd38/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
index 8b70185..dce48d1 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
@@ -1227,7 +1227,7 @@ public class AppState {
/**
* Is a role short lived by the threshold set for this application
* @param instance instance
- * @return true if the instance is considered short live
+ * @return true if the instance is considered short lived
*/
@VisibleForTesting
public boolean isShortLived(RoleInstance instance) {
@@ -1236,7 +1236,7 @@ public class AppState {
boolean shortlived;
if (started 0) {
long duration = time - started;
- shortlived = duration startTimeThreshold;
+ shortlived = duration (startTimeThreshold * 1000);
} else {
// never even saw a start event
shortlived = true;
[50/50] git commit: Merge branch 'develop' into feature/SLIDER-280_Restart_AM_fun_tests
Merge branch 'develop' into feature/SLIDER-280_Restart_AM_fun_tests Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/278d5e4c Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/278d5e4c Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/278d5e4c Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 278d5e4c7342fc43ccb4324008b3351e35ec7bcb Parents: 71021f0 930920d Author: Sumit Mohanty smoha...@hortonworks.com Authored: Thu Sep 4 12:38:21 2014 -0700 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Thu Sep 4 12:38:21 2014 -0700 -- README.md | 32 +- app-packages/accumulo/README.txt| 55 +- app-packages/accumulo/appConfig.json| 32 +- app-packages/accumulo/metainfo.xml | 16 + .../accumulo/package/scripts/accumulo_client.py | 2 +- .../package/scripts/accumulo_configuration.py | 64 +- .../accumulo/package/scripts/accumulo_script.py | 53 +- .../package/scripts/accumulo_service.py | 2 +- app-packages/accumulo/package/scripts/params.py | 28 +- .../package/templates/accumulo-env.sh.j2| 2 +- .../accumulo/package/templates/client.conf.j2 | 18 + app-packages/accumulo/pom.xml | 41 +- app-packages/accumulo/src/assembly/accumulo.xml | 24 +- .../slider/accumulo/CustomAuthenticator.java| 118 .../apache/slider/accumulo/ProviderUtil.java| 57 ++ .../AccumuloAgentCommandTestBase.groovy | 10 +- .../funtest/accumulo/AccumuloBasicIT.groovy | 96 ++- .../accumulo/AccumuloMonitorSSLIT.groovy| 2 +- .../funtest/accumulo/AccumuloReadWriteIT.groovy | 7 +- .../accumulo/AccumuloReadWriteSSLIT.groovy | 73 +++ .../funtest/accumulo/AccumuloSSLTestBase.groovy | 105 .../slider/funtest/accumulo/CertUtil.java | 235 +++ .../src/test/resources/appConfig_kerberos.json | 55 ++ .../test/resources/appConfig_monitor_ssl.json | 35 +- .../src/test/resources/appConfig_ssl.json | 55 ++ .../accumulo/src/test/resources/resources.json | 4 +- .../test/resources/resources_with_client.json | 41 ++ app-packages/app-pkg-template/appConfig.json| 2 +- app-packages/app-pkg-template/metainfo.xml | 8 +- .../command-logger/slider-pkg/appConfig.json| 2 - .../command-logger/slider-pkg/metainfo.xml | 8 + app-packages/hbase-win/README.txt | 34 + app-packages/hbase-win/appConfig.json | 44 ++ .../hbase-win/configuration/hbase-env.xml | 54 ++ .../hbase-win/configuration/hbase-log4j.xml | 143 + .../hbase-win/configuration/hbase-policy.xml| 53 ++ .../hbase-win/configuration/hbase-site.xml | 370 +++ app-packages/hbase-win/jmx_metrics.json | 56 ++ app-packages/hbase-win/metainfo.xml | 129 ...se-0.98.0.2.1.5.0-2047-hadoop2.zip.REPLACEME | 17 + .../hbase-win/package/scripts/__init__.py | 19 + .../hbase-win/package/scripts/functions.py | 40 ++ app-packages/hbase-win/package/scripts/hbase.py | 66 ++ .../hbase-win/package/scripts/hbase_master.py | 63 ++ .../package/scripts/hbase_regionserver.py | 63 ++ .../hbase-win/package/scripts/hbase_service.py | 48 ++ .../hbase-win/package/scripts/params.py | 81 +++ .../hbase-win/package/scripts/status_params.py | 26 + .../package/templates/hbase_client_jaas.conf.j2 | 22 + .../package/templates/hbase_master_jaas.conf.j2 | 25 + .../templates/hbase_regionserver_jaas.conf.j2 | 25 + app-packages/hbase-win/resources.json | 21 + app-packages/hbase/appConfig.json | 105 ++-- app-packages/hbase/configuration/global.xml | 160 - app-packages/hbase/configuration/hbase-env.xml | 117 .../hbase/configuration/hbase-log4j.xml | 1 + app-packages/hbase/configuration/hbase-site.xml | 19 +- app-packages/hbase/get-hbase-site.sh| 15 + app-packages/hbase/metainfo.xml | 24 + app-packages/hbase/package/scripts/hbase.py | 28 +- app-packages/hbase/package/scripts/params.py| 26 +- .../hbase/package/templates/hbase-env.sh.j2 | 81 --- app-packages/hbase/pom.xml | 4 +- app-packages/memcached-win/appConfig.json | 6 +- app-packages/memcached-win/metainfo.xml | 8 +- app-packages/memcached/appConfig.json | 4 +- app-packages/memcached/metainfo.xml | 8 +- app-packages/storm/README.txt | 19 +- app-packages/storm/appConfig.json | 103 +-- app-packages/storm/configuration/global.xml | 39 -- app-packages/storm/configuration/storm-env.xml | 65 ++ app-packages/storm/configuration/storm-site.xml | 15 +- app-packages/storm/metainfo.xml
[32/50] git commit: SLIDER-350 use gawk as awk implementation for windows; add checks for that and xargs on the command line
SLIDER-350 use gawk as awk implementation for windows; add checks for that and
xargs on the command line
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8f9caa9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8f9caa9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8f9caa9b
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 8f9caa9ba8ba5a09df794096d0096356535dce38
Parents: 690793a
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:01:30 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:01:30 2014 +0100
--
.../org/apache/slider/common/tools/TestWindowsSupport.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8f9caa9b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index d18fe76..61ede62 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -155,7 +155,7 @@ class TestWindowsSupport extends SliderTestBase {
* @param commands
* @return
*/
- public ForkedProcessService exec(ArrayListString commands) {
+ public ForkedProcessService exec(ListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
methodName.methodName,
[45/50] git commit: Merge branch 'feature/feature/SLIDER-377' into develop
Merge branch 'feature/feature/SLIDER-377' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a0a3ae63 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a0a3ae63 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a0a3ae63 Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: a0a3ae632fc34be0137ac71b1564ca319c757632 Parents: 88d72dd 4fc0396 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 18:19:08 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 18:19:08 2014 +0100 -- .../org/apache/slider/client/SliderClient.java | 6 ++- .../agent/actions/TestActionVersion.groovy | 5 +-- .../tools/TestClientResourceRegistration.groovy | 3 ++ .../common/tools/TestConfigHelperHDFS.groovy| 3 ++ .../common/tools/TestMiscSliderUtils.groovy | 3 ++ .../common/tools/TestZKIntegration.groovy | 44 ++-- .../slider/providers/agent/AgentTestBase.groovy | 2 +- .../slider/test/YarnMiniClusterTestBase.groovy | 27 ++-- 8 files changed, 52 insertions(+), 41 deletions(-) --
[28/50] git commit: SLIDER-181. Storm package updates
SLIDER-181. Storm package updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/38c816eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/38c816eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/38c816eb
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 38c816ebbd3c883308bbe36fc00846d4ff4dac7b
Parents: e9a353c
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Sep 3 13:15:20 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Sep 3 13:46:02 2014 -0700
--
app-packages/storm/README.txt | 19 ++---
app-packages/storm/appConfig.json | 10 +--
app-packages/storm/metainfo.xml | 2 +-
...pache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE | 16
app-packages/storm/pom.xml | 90
app-packages/storm/src/assembly/storm.xml | 68 +++
6 files changed, 172 insertions(+), 33 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38c816eb/app-packages/storm/README.txt
--
diff --git a/app-packages/storm/README.txt b/app-packages/storm/README.txt
index d4df85f..49adfce 100644
--- a/app-packages/storm/README.txt
+++ b/app-packages/storm/README.txt
@@ -17,19 +17,16 @@
How to create a Slider app package for Storm?
-To create the app package you will need the Storm tarball copied to a specific
location.
-Various configurations provided in this sample are customized for
apache-storm-0.9.3.2.2.0.0-578.tar.gz.
-So if you use a different version you may need to edit a few config values.
+To create the app package you will need the Storm tarball and invoke mvn
command
+with appropriate parameters.
-Replace the placeholder tarball for Storm.
- cp ~/Downloads/apache-storm-0.9.3.2.2.0.0-578.tar.gz package/files/
- rm package/files/apache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE
+Command:
+mvn clean package -Pstorm-app-package -Dpkg.version=version
+ -Dpkg.name=file name of app tarball -Dpkg.src=folder location where the
pkg is available
-Create a zip package at the root of the package (slider
enlistment/app-packages/storm/)
- zip -r apache-storm-0.9.3.zip .
-
-Verify the content using
- unzip -l $@ apache-storm-0.9.3.zip
+Example:
+mvn clean package -Pstorm-app-package -Dpkg.version=0.9.3.2.2.0.0-578
+ -Dpkg.name=apache-storm-0.9.3.2.2.0.0-578.tar.gz
-Dpkg.src=/Users/user1/Downloads
While appConfig.json and resources.json are not required for the package they
work
well as the default configuration for Slider apps. So its advisable that when
you
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38c816eb/app-packages/storm/appConfig.json
--
diff --git a/app-packages/storm/appConfig.json
b/app-packages/storm/appConfig.json
index b5fe436..df28e8f 100644
--- a/app-packages/storm/appConfig.json
+++ b/app-packages/storm/appConfig.json
@@ -3,13 +3,13 @@
metadata: {
},
global: {
-application.def: apache-storm-0.9.3.zip,
+application.def: slider-storm-app-package-${pkg.version}.zip,
java_home: /usr/jdk64/jdk1.7.0_45,
create.default.zookeeper.node: true,
system_configs: core-site,
site.global.app_user: yarn,
-site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version},
site.global.user_group: hadoop,
site.global.ganglia_server_host: ${NN_HOST},
site.global.ganglia_server_id: Application2,
@@ -22,8 +22,8 @@
site.storm-site.storm.local.dir: ${AGENT_WORK_ROOT}/app/tmp/storm,
site.storm-site.transactional.zookeeper.root: /transactional,
site.storm-site.storm.zookeeper.port: 2181,
-site.storm-site.nimbus.childopts: -Xmx1024m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM,
-site.storm-site.worker.childopts: -Xmx768m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM,
+site.storm-site.nimbus.childopts: -Xmx1024m
[21/50] git commit: SLIDER-387 adjusted default accumulo configuration
SLIDER-387 adjusted default accumulo configuration
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0ac2536f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0ac2536f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0ac2536f
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 0ac2536f96e9010569deb0a0d2079f8c23c17e06
Parents: 7ecab23
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Tue Sep 2 11:17:46 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Tue Sep 2 11:17:46 2014 -0700
--
app-packages/accumulo/appConfig.json| 5 +++--
.../accumulo/src/test/resources/appConfig_monitor_ssl.json | 5 +++--
app-packages/accumulo/src/test/resources/appConfig_ssl.json | 5 +++--
3 files changed, 9 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ac2536f/app-packages/accumulo/appConfig.json
--
diff --git a/app-packages/accumulo/appConfig.json
b/app-packages/accumulo/appConfig.json
index a44dbad..2c0c2d5 100644
--- a/app-packages/accumulo/appConfig.json
+++ b/app-packages/accumulo/appConfig.json
@@ -9,7 +9,7 @@
site.global.app_log_dir: ${AGENT_LOG_ROOT},
site.global.app_pid_dir: ${AGENT_WORK_ROOT}/app/run,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/accumulo-${accumulo.version},
-site.global.tserver_heapsize: 128m,
+site.global.tserver_heapsize: 256m,
site.global.master_heapsize: 128m,
site.global.monitor_heapsize: 64m,
site.global.gc_heapsize: 64m,
@@ -27,12 +27,13 @@
site.accumulo-site.instance.zookeeper.host: ${ZK_HOST},
site.accumulo-site.instance.security.authenticator:
org.apache.slider.accumulo.CustomAuthenticator,
site.accumulo-site.general.security.credential.provider.paths:
jceks://hdfs/user/${USER}/accumulo-${CLUSTER_NAME}.jceks,
+site.accumulo-site.tserver.memory.maps.native.enabled: false,
site.accumulo-site.tserver.memory.maps.max: 80M,
site.accumulo-site.tserver.cache.data.size: 7M,
site.accumulo-site.tserver.cache.index.size: 20M,
site.accumulo-site.trace.user: root,
site.accumulo-site.tserver.sort.buffer.size: 50M,
-site.accumulo-site.tserver.walog.max.size: 100M,
+site.accumulo-site.tserver.walog.max.size: 40M,
site.accumulo-site.master.port.client: 0,
site.accumulo-site.trace.port.client: 0,
site.accumulo-site.tserver.port.client: 0,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ac2536f/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
--
diff --git
a/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
b/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
index c6a2639..45e6e19 100644
--- a/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
+++ b/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
@@ -9,7 +9,7 @@
site.global.app_log_dir: ${AGENT_LOG_ROOT},
site.global.app_pid_dir: ${AGENT_WORK_ROOT}/app/run,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/accumulo-${accumulo.version},
-site.global.tserver_heapsize: 128m,
+site.global.tserver_heapsize: 256m,
site.global.master_heapsize: 128m,
site.global.monitor_heapsize: 64m,
site.global.gc_heapsize: 64m,
@@ -27,12 +27,13 @@
site.accumulo-site.instance.zookeeper.host: ${ZK_HOST},
site.accumulo-site.instance.security.authenticator:
org.apache.slider.accumulo.CustomAuthenticator,
site.accumulo-site.general.security.credential.provider.paths:
jceks://hdfs/user/${USER}/accumulo-${CLUSTER_NAME}.jceks,
+site.accumulo-site.tserver.memory.maps.native.enabled: false,
site.accumulo-site.tserver.memory.maps.max: 80M,
site.accumulo-site.tserver.cache.data.size: 7M,
site.accumulo-site.tserver.cache.index.size: 20M,
site.accumulo-site.trace.user: root,
site.accumulo-site.tserver.sort.buffer.size: 50M,
-site.accumulo-site.tserver.walog.max.size: 100M,
+site.accumulo-site.tserver.walog.max.size: 40M,
site.accumulo-site.master.port.client: 0,
site.accumulo-site.trace.port.client: 0,
site.accumulo-site.tserver.port.client: 0,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ac2536f/app-packages/accumulo/src/test/resources/appConfig_ssl.json
--
diff --git a/app-packages/accumulo/src/test/resources/appConfig_ssl.json
b/app-packages/accumulo/src/test/resources/appConfig_ssl.json
index f48216d..3df 100644
---
[44/50] git commit: SLIDER-377 tuning of some tests making a couple of methods accessible (groovy was apparently bending the rules)
SLIDER-377 tuning of some tests making a couple of methods accessible (groovy
was apparently bending the rules)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/4fc0396e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/4fc0396e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/4fc0396e
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 4fc0396e8705b0e50bbc6bd9671c1b8b5dbb3a0a
Parents: f2965d8
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 18:19:01 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 18:19:01 2014 +0100
--
.../org/apache/slider/client/SliderClient.java | 6 ++-
.../agent/actions/TestActionVersion.groovy | 5 +--
.../common/tools/TestZKIntegration.groovy | 44 ++--
.../slider/providers/agent/AgentTestBase.groovy | 2 +-
.../slider/test/YarnMiniClusterTestBase.groovy | 27 ++--
5 files changed, 43 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4fc0396e/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index f4040c4..ef6448d 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -355,7 +355,8 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
/**
* Delete the zookeeper node associated with the calling user and the cluster
**/
- protected boolean deleteZookeeperNode(String clusterName) throws
YarnException, IOException {
+ @VisibleForTesting
+ public boolean deleteZookeeperNode(String clusterName) throws YarnException,
IOException {
String user = getUsername();
String zkPath = ZKIntegration.mkClusterPath(user, clusterName);
Exception e = null;
@@ -391,7 +392,8 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
/**
* Create the zookeeper node associated with the calling user and the cluster
*/
- protected String createZookeeperNode(String clusterName, Boolean nameOnly)
throws YarnException, IOException {
+ @VisibleForTesting
+ public String createZookeeperNode(String clusterName, Boolean nameOnly)
throws YarnException, IOException {
String user = getUsername();
String zkPath = ZKIntegration.mkClusterPath(user, clusterName);
if(nameOnly) {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4fc0396e/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
index 1d50b71..7a2820f 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/actions/TestActionVersion.groovy
@@ -20,10 +20,9 @@ package org.apache.slider.agent.actions
import groovy.util.logging.Slf4j
import org.apache.slider.common.params.SliderActions
-import org.apache.slider.test.YarnMiniClusterTestBase
+import org.apache.slider.test.SliderTestBase
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.apache.slider.core.main.ServiceLauncher
-import org.junit.Before
import org.junit.Test
/**
@@ -31,7 +30,7 @@ import org.junit.Test
*/
@Slf4j
-class TestActionVersion extends YarnMiniClusterTestBase {
+class TestActionVersion extends SliderTestBase {
@Test
public void testVersion() throws Throwable {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4fc0396e/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
index fe3bef7..c13af78 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
@@ -18,6 +18,7 @@
package org.apache.slider.common.tools
+import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
import org.apache.slider.client.SliderClient
@@
[08/50] git commit: SLIDER-375 path fixup for windows
SLIDER-375 path fixup for windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/aceab6ba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/aceab6ba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/aceab6ba
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: aceab6ba942fb5e6c338570e030b4e09dce609eb
Parents: 26ec0d0
Author: Steve Loughran ste...@apache.org
Authored: Fri Aug 29 12:42:28 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Fri Aug 29 13:19:00 2014 +0100
--
.../slider/providers/agent/TestAgentAMManagementWS.groovy | 8 +---
.../apache/slider/providers/agent/TestAgentEcho.groovy| 6 +++---
.../registry/curator/TestRegistryRestResources.groovy | 6 +++---
.../web/rest/publisher/TestPublisherRestResources.groovy | 10 +++---
.../groovy/org/apache/slider/test/SliderTestUtils.groovy | 9 +
5 files changed, 23 insertions(+), 16 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/aceab6ba/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
index 4e7b5b9..6720bda 100644
---
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
@@ -117,9 +117,9 @@ class TestAgentAMManagementWS extends AgentTestBase {
roles,
[
ARG_OPTION, PACKAGE_PATH, slider_core.absolutePath,
-ARG_OPTION, APP_DEF, app_def_path.absoluteFile.toURI().toString(),
-ARG_OPTION, AGENT_CONF,
agt_conf_path.absoluteFile.toURI().toString(),
-ARG_OPTION, AGENT_VERSION,
agt_ver_path.absoluteFile.toURI().toString(),
+ARG_OPTION, APP_DEF, toURIArg(app_def_path),
+ARG_OPTION, AGENT_CONF, toURIArg(agt_conf_path),
+ARG_OPTION, AGENT_VERSION, toURIArg(agt_ver_path),
],
true, true,
true)
@@ -163,4 +163,6 @@ class TestAgentAMManagementWS extends AgentTestBase {
assert RegistrationStatus.FAILED == response.getResponseStatus();
}
+
+
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/aceab6ba/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentEcho.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentEcho.groovy
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentEcho.groovy
index a29c8cb..a7b5fe3 100644
---
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentEcho.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentEcho.groovy
@@ -72,9 +72,9 @@ class TestAgentEcho extends AgentTestBase {
roles,
[
ARG_OPTION, PACKAGE_PATH, slider_core.absolutePath,
-ARG_OPTION, APP_DEF, file:// + app_def_path.absolutePath,
-ARG_OPTION, AGENT_CONF, file:// + agt_conf_path.absolutePath,
-ARG_OPTION, AGENT_VERSION, file:// + agt_ver_path.absolutePath,
+ARG_OPTION, APP_DEF, toURIArg(app_def_path),
+ARG_OPTION, AGENT_CONF, toURIArg(agt_conf_path),
+ARG_OPTION, AGENT_VERSION, toURIArg(agt_ver_path),
ARG_RES_COMP_OPT, role, ResourceKeys.COMPONENT_PRIORITY, 1,
ARG_COMP_OPT, role, SCRIPT_PATH, echo_py,
ARG_COMP_OPT, role, SERVICE_NAME, Agent,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/aceab6ba/slider-core/src/test/groovy/org/apache/slider/registry/curator/TestRegistryRestResources.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/registry/curator/TestRegistryRestResources.groovy
b/slider-core/src/test/groovy/org/apache/slider/registry/curator/TestRegistryRestResources.groovy
index 1a1e5aa..0b77adc 100644
---
a/slider-core/src/test/groovy/org/apache/slider/registry/curator/TestRegistryRestResources.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/registry/curator/TestRegistryRestResources.groovy
@@ -88,9 +88,9 @@ class TestRegistryRestResources extends AgentTestBase {
roles,
[
ARG_OPTION, PACKAGE_PATH, slider_core.absolutePath,
-ARG_OPTION, APP_DEF, file:// + app_def_path.absolutePath,
-ARG_OPTION, AGENT_CONF,
[10/50] git commit: Merge branch 'feature/SLIDER-375_slider-core_tests_failing_on_windows' into develop
Merge branch 'feature/SLIDER-375_slider-core_tests_failing_on_windows' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/c91d5ecd Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/c91d5ecd Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/c91d5ecd Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: c91d5ecd3a8b78d218bd02a5e2e7c9e7fefbc8cf Parents: 8bfd66e aceab6b Author: Steve Loughran ste...@apache.org Authored: Fri Aug 29 13:19:52 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Fri Aug 29 13:19:52 2014 +0100 -- .../agent/TestAgentAMManagementWS.groovy| 8 +++--- .../slider/providers/agent/TestAgentEcho.groovy | 6 ++--- .../curator/TestRegistryRestResources.groovy| 6 ++--- .../publisher/TestPublisherRestResources.groovy | 26 +--- .../apache/slider/test/SliderTestUtils.groovy | 9 +++ .../workflow/ProcessCommandFactory.java | 21 +++- .../services/workflow/TestLongLivedProcess.java | 6 ++--- .../workflow/WorkflowServiceTestBase.java | 3 ++- 8 files changed, 51 insertions(+), 34 deletions(-) --
[41/50] git commit: SLIDER-377 AM-spawning tests do prelaunch validity checks
SLIDER-377 AM-spawning tests do prelaunch validity checks
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2d6f276d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2d6f276d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2d6f276d
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 2d6f276d16cc93648ccf7306eb94f5ac09bb5763
Parents: 0164277
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 17:09:12 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 17:09:12 2014 +0100
--
.../org/apache/slider/agent/AgentMiniClusterTestBase.groovy | 9 +
1 file changed, 9 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2d6f276d/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
index 4566d9d..7786c41 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/AgentMiniClusterTestBase.groovy
@@ -27,6 +27,7 @@ import org.apache.commons.io.FileUtils
import org.apache.slider.client.SliderClient
import org.apache.slider.common.SliderXMLConfKeysForTesting
import org.apache.slider.common.params.Arguments
+import org.apache.slider.common.tools.SliderUtils
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.providers.agent.AgentKeys
import org.apache.slider.test.YarnZKMiniClusterTestBase
@@ -47,6 +48,14 @@ extends YarnZKMiniClusterTestBase {
protected static MapString, String agentDefOptions
private static TemporaryFolder tempFolder = new TemporaryFolder();
+ /**
+ * Server side test: validate system env before launch
+ */
+ @BeforeClass
+ public static void checkSystem() {
+SliderUtils.validateSliderServerEnvironment(null)
+ }
+
@BeforeClass
public static void createSubConfFiles() {
[20/50] git commit: SLIDER-386 fixed role-specific failure threshold
SLIDER-386 fixed role-specific failure threshold
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/7ecab238
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/7ecab238
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/7ecab238
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 7ecab238a919ae4013f23b259ab4eba317635485
Parents: 82cf1f0
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Tue Sep 2 08:48:10 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Tue Sep 2 08:48:10 2014 -0700
--
.../java/org/apache/slider/server/appmaster/state/AppState.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7ecab238/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
index 07976ef..8b70185 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
@@ -1559,7 +1559,7 @@ public class AppState {
ConfTreeOperations resources =
instanceDefinition.getResourceOperations();
return resources.getComponentOptInt(roleStatus.getName(),
-ResourceKeys.CONTAINER_FAILURE_SHORTLIFE,
+ResourceKeys.CONTAINER_FAILURE_THRESHOLD,
failureThreshold);
}
[11/50] git commit: SLIDER-377: changing outcome if winutils.exe is not on the path
SLIDER-377: changing outcome if winutils.exe is not on the path
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/686fdff3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/686fdff3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/686fdff3
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 686fdff358a978fdc56240e82c045dfc5ae0a591
Parents: c91d5ec
Author: Steve Loughran ste...@apache.org
Authored: Fri Aug 29 17:54:21 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Fri Aug 29 17:54:21 2014 +0100
--
.../org/apache/slider/test/YarnMiniClusterTestBase.groovy | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/686fdff3/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
index 5a091bf..5aef585 100644
---
a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
@@ -132,7 +132,12 @@ public abstract class YarnMiniClusterTestBase extends
ServiceLauncherBaseTest {
@BeforeClass
public static void checkWindowsSupport() {
if (Shell.WINDOWS) {
- assertNotNull(winutils.exe not found, Shell.WINUTILS)
+// assertNotNull(winutils.exe not found, Shell.WINUTILS)
+ if (!Shell.WINUTILS) {
+log.error(winutils.exe not found)
+ }
+ def lib = System.getProperty(java.library.path)
+ log.debug(java.library.path = ${lib})
}
}
[25/50] git commit: SLIDER-181. Storm package updates
SLIDER-181. Storm package updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/201686e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/201686e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/201686e2
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 201686e2ea76d6f7f80554e5b9ad9471fce152be
Parents: e7dd217
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Sep 2 18:43:53 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Sep 2 18:43:53 2014 -0700
--
app-packages/storm/README.txt | 12 -
app-packages/storm/appConfig.json | 15 +--
app-packages/storm/configuration/storm-env.xml | 25 ++
app-packages/storm/metainfo.xml | 26 +++
...pache-storm-0.9.1.2.1.1.0-237.tar.gz.REPLACE | 16
...pache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE | 16
app-packages/storm/package/scripts/params.py| 15 ++-
app-packages/storm/package/scripts/service.py | 8 ++
.../storm/package/scripts/status_params.py | 4 +--
.../storm/package/templates/config.yaml.j2 | 9 ---
.../storm/package/templates/storm_jaas.conf.j2 | 27 +---
app-packages/storm/resources.json | 10 +++-
12 files changed, 101 insertions(+), 82 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/201686e2/app-packages/storm/README.txt
--
diff --git a/app-packages/storm/README.txt b/app-packages/storm/README.txt
index 971cf14..d4df85f 100644
--- a/app-packages/storm/README.txt
+++ b/app-packages/storm/README.txt
@@ -18,18 +18,18 @@
How to create a Slider app package for Storm?
To create the app package you will need the Storm tarball copied to a specific
location.
-Various configurations provided in this sample are customized for
apache-storm-0.9.1.2.1.1.0-237.tar.gz.
+Various configurations provided in this sample are customized for
apache-storm-0.9.3.2.2.0.0-578.tar.gz.
So if you use a different version you may need to edit a few config values.
Replace the placeholder tarball for Storm.
- cp ~/Downloads/apache-storm-0.9.1.2.1.1.0-237.tar.gz package/files/
- rm package/files/apache-storm-0.9.1.2.1.1.0-237.tar.gz.REPLACE
+ cp ~/Downloads/apache-storm-0.9.3.2.2.0.0-578.tar.gz package/files/
+ rm package/files/apache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE
-Create a zip package at the root of the package (slider
enlistment/app-packages/storm-v0_91/)
- zip -r storm_v091.zip .
+Create a zip package at the root of the package (slider
enlistment/app-packages/storm/)
+ zip -r apache-storm-0.9.3.zip .
Verify the content using
- unzip -l $@ storm_v091.zip
+ unzip -l $@ apache-storm-0.9.3.zip
While appConfig.json and resources.json are not required for the package they
work
well as the default configuration for Slider apps. So its advisable that when
you
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/201686e2/app-packages/storm/appConfig.json
--
diff --git a/app-packages/storm/appConfig.json
b/app-packages/storm/appConfig.json
index b8454f9..b5fe436 100644
--- a/app-packages/storm/appConfig.json
+++ b/app-packages/storm/appConfig.json
@@ -3,28 +3,27 @@
metadata: {
},
global: {
-application.def: storm_v091.zip,
+application.def: apache-storm-0.9.3.zip,
java_home: /usr/jdk64/jdk1.7.0_45,
create.default.zookeeper.node: true,
+system_configs: core-site,
site.global.app_user: yarn,
-site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578,
site.global.user_group: hadoop,
-site.global.security_enabled: false,
site.global.ganglia_server_host: ${NN_HOST},
site.global.ganglia_server_id: Application2,
site.global.ganglia_enabled:true,
site.global.ganglia_server_port: 8668,
-site.global.rest_api_port: ${STORM_REST_API.ALLOCATED_PORT},
-site.global.rest_api_admin_port: ${STORM_REST_API.ALLOCATED_PORT},
+site.storm-site.storm.log.dir : ${AGENT_LOG_ROOT},
site.storm-site.storm.zookeeper.servers: ['${ZK_HOST}'],
site.storm-site.nimbus.thrift.port: ${NIMBUS.ALLOCATED_PORT},
site.storm-site.storm.local.dir: ${AGENT_WORK_ROOT}/app/tmp/storm,
site.storm-site.transactional.zookeeper.root: /transactional,
site.storm-site.storm.zookeeper.port: 2181,
-site.storm-site.nimbus.childopts: -Xmx1024m
[12/50] git commit: using is None over == None in .py
using is None over == None in .py Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/329cab03 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/329cab03 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/329cab03 Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 329cab03e07759fdaea4d77737abe0a42d4f9947 Parents: 686fdff Author: Steve Loughran ste...@apache.org Authored: Fri Aug 29 19:09:07 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Fri Aug 29 19:09:07 2014 +0100 -- slider-assembly/src/main/scripts/slider.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/329cab03/slider-assembly/src/main/scripts/slider.py -- diff --git a/slider-assembly/src/main/scripts/slider.py b/slider-assembly/src/main/scripts/slider.py index db4b881..be79973 100644 --- a/slider-assembly/src/main/scripts/slider.py +++ b/slider-assembly/src/main/scripts/slider.py @@ -147,9 +147,9 @@ def java(classname, args, classpath, jvm_opts_list): # split the JVM opts by space # java = /usr/bin/java - prg=java - if which(java)==None: -prg=os.environ[JAVA_HOME]+/bin/java + prg = java + if which(java) is None: +prg = os.environ[JAVA_HOME] + /bin/java commandline = [prg] commandline.extend(jvm_opts_list) commandline.append(-classpath)
[39/50] git commit: SLIDER-377 AM checks for validity of server side; client does this if the CLI action says it should (default is yes)
SLIDER-377 AM checks for validity of server side; client does this if the CLI
action says it should (default is yes)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b6361803
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b6361803
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b6361803
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: b636180392a719b1b663f77592c23b7b16a8f4d3
Parents: 8034012
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:53:35 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:53:35 2014 +0100
--
.../org/apache/slider/client/SliderClient.java| 18 ++
.../slider/common/params/AbstractActionArgs.java | 11 +++
.../slider/common/params/ActionHelpArgs.java | 14 --
.../slider/common/params/ActionVersionArgs.java | 12 +++-
.../apache/slider/common/tools/SliderUtils.java | 2 +-
.../apache/slider/core/build/InstanceBuilder.java | 5 +++--
.../slider/server/appmaster/SliderAppMaster.java | 2 +-
.../services/workflow/LongLivedProcess.java | 3 ++-
8 files changed, 55 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b6361803/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index f6d761e..f4040c4 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -46,6 +46,7 @@ import org.apache.slider.api.proto.Messages;
import org.apache.slider.common.Constants;
import org.apache.slider.common.SliderExitCodes;
import org.apache.slider.common.SliderKeys;
+import org.apache.slider.common.params.AbstractActionArgs;
import org.apache.slider.common.params.AbstractClusterBuildingActionArgs;
import org.apache.slider.common.params.ActionAMSuicideArgs;
import org.apache.slider.common.params.ActionCreateArgs;
@@ -291,6 +292,12 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
// choose the action
String action = serviceArgs.getAction();
+
+AbstractActionArgs coreAction = serviceArgs.getCoreAction();
+if (coreAction.getHadoopServicesRequired()) {
+ // validate the client
+ SliderUtils.validateSliderClientEnvironment(null);
+}
int exitCode = EXIT_SUCCESS;
String clusterName = serviceArgs.getClusterName();
// actions
@@ -613,7 +620,9 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
// verify that a live cluster isn't there
SliderUtils.validateClusterName(clustername);
verifyBindingsDefined();
-if (!liveClusterAllowed) verifyNoLiveClusters(clustername);
+if (!liveClusterAllowed) {
+ verifyNoLiveClusters(clustername);
+}
Configuration conf = getConfig();
String registryQuorum = lookupZKQuorum();
@@ -1169,7 +1178,7 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
*/
addConfOptionToCLI(commandLine,
config,
- DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
+ DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY);
}
// write out the path output
commandLine.addOutAndErrFiles(STDOUT_AM, STDERR_AM);
@@ -1263,10 +1272,11 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
*/
private void propagatePrincipals(Configuration config,
AggregateConf clusterSpec) {
-String dfsPrincipal = config.get(DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
+String dfsPrincipal = config.get(
+DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY);
if (dfsPrincipal != null) {
String siteDfsPrincipal = OptionKeys.SITE_XML_PREFIX +
-DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY;
+
DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY;
clusterSpec.getAppConfOperations().getGlobalOptions().putIfUnset(
siteDfsPrincipal,
dfsPrincipal);
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b6361803/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
b/slider-core/src/main/java/org/apache/slider/common/params/AbstractActionArgs.java
index
[29/50] git commit: SLIDER-350 switch to hadoop 3.0.0 for testing
SLIDER-350 switch to hadoop 3.0.0 for testing Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/bfdec589 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/bfdec589 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/bfdec589 Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: bfdec5897997511cef405885bd940da01a5189e7 Parents: 38c816e Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 12:59:28 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 12:59:28 2014 +0100 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bfdec589/pom.xml -- diff --git a/pom.xml b/pom.xml index 8292073..454918c 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,7 @@ !-- core artifacts -- -hadoop.version2.6.0-SNAPSHOT/hadoop.version +hadoop.version3.0.0-SNAPSHOT/hadoop.version hbase.version0.98.4-hadoop2/hbase.version accumulo.version1.6.1-SNAPSHOT/accumulo.version
[30/50] git commit: SLIDER-350 add strict test for windows utils being valid
SLIDER-350 add strict test for windows utils being valid
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2e3c4c81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2e3c4c81
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2e3c4c81
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 2e3c4c81b799cbf6d06e394c2b17f8f161f68734
Parents: bfdec58
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 13:46:59 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 13:46:59 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 50 +++-
.../common/tools/TestWindowsSupport.groovy | 30 ++--
2 files changed, 76 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2e3c4c81/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index a864878..eb214db 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -37,6 +37,7 @@ import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Shell;
import org.apache.hadoop.util.VersionInfo;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.Container;
@@ -63,6 +64,7 @@ import org.slf4j.LoggerFactory;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
@@ -104,8 +106,9 @@ public final class SliderUtils {
public static final String JAVA_SECURITY_KRB5_REALM =
java.security.krb5.realm;
public static final String JAVA_SECURITY_KRB5_KDC = java.security.krb5.kdc;
+ public static final String WINUTILS = WINUTILS.EXE;
+
-
private SliderUtils() {
}
@@ -1499,4 +1502,49 @@ public final class SliderUtils {
return is;
}
+
+ /**
+ * Strictly verify that windows utils is present.
+ * Checks go as far as opening the file and looking for
+ * the headers.
+ * @throws IOException on any problem reading the file
+ * @throws FileNotFoundException if the file is not considered valid
+ */
+ public static void verifyWinUtilsValid() throws IOException {
+if (!Shell.WINDOWS) {
+ return;
+}
+String winUtilsPath = Shell.getWinUtilsPath();
+if (winUtilsPath == null) {
+ throw new FileNotFoundException(WINUTILS + not found on Path : +
+ System.getenv(Path));
+}
+File winUtils = new File(winUtilsPath);
+if (!winUtils.isFile()) {
+ throw new FileNotFoundException(WINUTILS
+ + at + winUtilsPath
+ + is not a file);
+
+}
+if (winUtils.length() 0x100) {
+ throw new FileNotFoundException(WINUTILS
++ at + winUtilsPath
++ is too short to be an executable);
+}
+// now read two bytes and verify the header.
+FileReader reader = null;
+try {
+ int[] header = new int[2];
+ reader = new FileReader(winUtilsPath);
+ header[0] = reader.read();
+ header[1] = reader.read();
+ if (header[0] != 'M' || header[1] != 'Z') {
+throw new FileNotFoundException(WINUTILS
+ + at + winUtilsPath
+ + is not a windows executable file);
+ }
+} finally {
+ IOUtils.closeStream(reader);
+}
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2e3c4c81/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index 7f5edf3..84356ea 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -27,14 +27,16 @@ import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.FileSystem as HadoopFS
import org.apache.hadoop.util.Shell
import org.apache.slider.providers.agent.AgentUtils
-import
[09/50] git commit: SLIDER-375 test getting its args wrong on windows
SLIDER-375 test getting its args wrong on windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/26ec0d08
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/26ec0d08
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/26ec0d08
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 26ec0d084a16a7a6c5d27f946fa48451047263aa
Parents: c40e480
Author: Steve Loughran ste...@apache.org
Authored: Fri Aug 29 12:25:19 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Fri Aug 29 13:19:00 2014 +0100
--
.../slider/providers/agent/TestAgentAMManagementWS.groovy | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/26ec0d08/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
index 1cd8de6..4e7b5b9 100644
---
a/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/providers/agent/TestAgentAMManagementWS.groovy
@@ -117,9 +117,9 @@ class TestAgentAMManagementWS extends AgentTestBase {
roles,
[
ARG_OPTION, PACKAGE_PATH, slider_core.absolutePath,
-ARG_OPTION, APP_DEF, file:// + app_def_path.absolutePath,
-ARG_OPTION, AGENT_CONF, file:// + agt_conf_path.absolutePath,
-ARG_OPTION, AGENT_VERSION, file:// + agt_ver_path.absolutePath,
+ARG_OPTION, APP_DEF, app_def_path.absoluteFile.toURI().toString(),
+ARG_OPTION, AGENT_CONF,
agt_conf_path.absoluteFile.toURI().toString(),
+ARG_OPTION, AGENT_VERSION,
agt_ver_path.absoluteFile.toURI().toString(),
],
true, true,
true)
[31/50] git commit: SLIDER-350 use gawk as awk implementation for windows; add checks for that and xargs on the command line
SLIDER-350 use gawk as awk implementation for windows; add checks for that and
xargs on the command line
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/690793a4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/690793a4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/690793a4
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 690793a4f8a82a3b7395271803ae9af89eb2cdda
Parents: 2e3c4c8
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 14:00:01 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 14:00:01 2014 +0100
--
.../common/tools/TestWindowsSupport.groovy | 38 ++--
.../apache/slider/test/SliderTestUtils.groovy | 2 +-
2 files changed, 37 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/690793a4/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
index 84356ea..d18fe76 100644
---
a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestWindowsSupport.groovy
@@ -105,12 +105,33 @@ class TestWindowsSupport extends SliderTestBase {
}
+
+ @Test
+ public void testHasGawkInstalled() throws Throwable {
+assume(Shell.WINDOWS, not windows)
+assert 0 == exec([gawk, --version])
+ }
+
+ @Test
+ public void testHasXargsInstalled() throws Throwable {
+assume(Shell.WINDOWS, not windows)
+assert 0 == exec([xargs, --version])
+ }
+
+
@Test
public void testEmitKillCommand() throws Throwable {
killJavaProcesses(regionserver, 9)
}
@Test
+ public void testHadoopHomeDefined() throws Throwable {
+assume(Shell.WINDOWS, not windows)
+def hadoopHome = Shell.hadoopHome
+log.info(HADOOP_HOME=$hadoopHome)
+ }
+
+ @Test
public void testHasWinutils() throws Throwable {
assume(Shell.WINDOWS, not windows)
SliderUtils.verifyWinUtilsValid()
@@ -123,14 +144,27 @@ class TestWindowsSupport extends SliderTestBase {
assert winUtilsPath
File winUtils = new File(winUtilsPath)
log.debug(Winutils is at $winUtils))
+
+ForkedProcessService process = exec([winUtilsPath, systeminfo])
+assert 0 == process.exitCode
+ }
+
+
+ /**
+ * Exec a set of commands, wait a few seconds for it to finish.
+ * @param commands
+ * @return
+ */
+ public ForkedProcessService exec(ArrayListString commands) {
ForkedProcessService process;
process = new ForkedProcessService(
-methodName.methodName,
+methodName.methodName,
[:],
-[winUtilsPath, systeminfo]);
+commands);
process.init(new Configuration());
EndOfServiceWaiter waiter = new EndOfServiceWaiter(process);
process.start();
waiter.waitForServiceToStop(5000);
+process
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/690793a4/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
b/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
index 58a76af..89f1a34 100644
--- a/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
@@ -755,7 +755,7 @@ class SliderTestUtils extends Assert {
/*
jps -l | grep String | awk {print $1} | xargs -n 1 taskkill /PID
*/
- GString killCommand = \jps -l | grep \${grepString}\ | awk \{print
\$1}\ | xargs -n 1 taskkill /PID\
+ GString killCommand = \jps -l | grep \${grepString}\ | gawk \{print
\$1}\ | xargs -n 1 taskkill /PID\
commandString = [CMD, /C, killCommand]
}
Process command = commandString.execute()
[07/50] git commit: SLIDER-375 TestLongLivedProcess needed windows command translation
SLIDER-375 TestLongLivedProcess needed windows command translation
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6797222c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6797222c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6797222c
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 6797222c92b887ffadb066251b5886787bb4cf44
Parents: 8bfd66e
Author: Steve Loughran ste...@apache.org
Authored: Thu Aug 28 19:15:14 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Fri Aug 29 13:18:59 2014 +0100
--
.../workflow/ProcessCommandFactory.java | 21 ++--
.../services/workflow/TestLongLivedProcess.java | 6 ++
.../workflow/WorkflowServiceTestBase.java | 3 ++-
3 files changed, 19 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6797222c/slider-core/src/test/java/org/apache/slider/server/services/workflow/ProcessCommandFactory.java
--
diff --git
a/slider-core/src/test/java/org/apache/slider/server/services/workflow/ProcessCommandFactory.java
b/slider-core/src/test/java/org/apache/slider/server/services/workflow/ProcessCommandFactory.java
index 45fdc86..4a19417 100644
---
a/slider-core/src/test/java/org/apache/slider/server/services/workflow/ProcessCommandFactory.java
+++
b/slider-core/src/test/java/org/apache/slider/server/services/workflow/ProcessCommandFactory.java
@@ -18,8 +18,11 @@
package org.apache.slider.server.services.workflow;
+import org.apache.hadoop.util.Shell;
+
import java.io.File;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
/**
@@ -37,10 +40,12 @@ public class ProcessCommandFactory {
* @return commands
*/
public ListString ls(File dir) {
-ListString commands = new ArrayListString(5);
-commands.add(ls);
-commands.add(-1);
-commands.add(dir.getAbsolutePath());
+ListString commands;
+if (!Shell.WINDOWS) {
+ commands = Arrays.asList(ls,-1, dir.getAbsolutePath());
+} else {
+ commands = Arrays.asList(cmd, /c, dir, dir.getAbsolutePath());
+}
return commands;
}
@@ -61,8 +66,12 @@ public class ProcessCommandFactory {
* @return commands
*/
public ListString env() {
-ListString commands = new ArrayListString(1);
-commands.add(env);
+ListString commands;
+if (!Shell.WINDOWS) {
+ commands = Arrays.asList(env);
+} else {
+ commands = Arrays.asList(cmd, /c, set);
+}
return commands;
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6797222c/slider-core/src/test/java/org/apache/slider/server/services/workflow/TestLongLivedProcess.java
--
diff --git
a/slider-core/src/test/java/org/apache/slider/server/services/workflow/TestLongLivedProcess.java
b/slider-core/src/test/java/org/apache/slider/server/services/workflow/TestLongLivedProcess.java
index 668bcca..9019124 100644
---
a/slider-core/src/test/java/org/apache/slider/server/services/workflow/TestLongLivedProcess.java
+++
b/slider-core/src/test/java/org/apache/slider/server/services/workflow/TestLongLivedProcess.java
@@ -33,11 +33,9 @@ import java.util.List;
*/
public class TestLongLivedProcess extends WorkflowServiceTestBase implements
LongLivedProcessLifecycleEvent {
- private static final Logger
- log = LoggerFactory.getLogger(TestLongLivedProcess.class);
+ private static final Logger log =
LoggerFactory.getLogger(TestLongLivedProcess.class);
- private static final Logger
- processLog =
+ private static final Logger processLog =
LoggerFactory.getLogger(org.apache.hadoop.services.workflow.Process);
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6797222c/slider-core/src/test/java/org/apache/slider/server/services/workflow/WorkflowServiceTestBase.java
--
diff --git
a/slider-core/src/test/java/org/apache/slider/server/services/workflow/WorkflowServiceTestBase.java
b/slider-core/src/test/java/org/apache/slider/server/services/workflow/WorkflowServiceTestBase.java
index 3049d8f..f38bd9d 100644
---
a/slider-core/src/test/java/org/apache/slider/server/services/workflow/WorkflowServiceTestBase.java
+++
b/slider-core/src/test/java/org/apache/slider/server/services/workflow/WorkflowServiceTestBase.java
@@ -29,6 +29,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
+import java.util.Locale;
import java.util.concurrent.Callable;
/**
@@ -122,7 +123,7 @@ public abstract class WorkflowServiceTestBase extends
Assert {
[46/50] git commit: SLIDER-393 fix pom to publish jars to ASF staging and snapshots (Giridharan Kesavan via billie)
SLIDER-393 fix pom to publish jars to ASF staging and snapshots (Giridharan
Kesavan via billie)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6126db62
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6126db62
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6126db62
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 6126db62bb5d4ae45b6ecddb7b71ce1b8ae13cb2
Parents: a0a3ae6
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Thu Sep 4 10:40:03 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Thu Sep 4 10:40:03 2014 -0700
--
pom.xml | 41 +++--
1 file changed, 39 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6126db62/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8292073..9efb5c8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -72,6 +72,19 @@
urlhttp://slider.incubator.apache.org//url
/site
downloadUrlhttp://git-wip-us.apache.org/repos/asf/incubator-slider.git/downloadUrl
+repository
+ id${distMgmtStagingId}/id
+ name${distMgmtStagingName}/name
+ url${distMgmtStagingUrl}/url
+/repository
+snapshotRepository
+ id${distMgmtSnapshotsId}/id
+ name${distMgmtSnapshotsName}/name
+ url${distMgmtSnapshotsUrl}/url
+/snapshotRepository
+
+
+
/distributionManagement
mailingLists
@@ -92,7 +105,12 @@
/mailingLists
properties
-
+distMgmtSnapshotsIdapache.snapshots.https/distMgmtSnapshotsId
+distMgmtSnapshotsNameApache Development Snapshot
Repository/distMgmtSnapshotsName
+
distMgmtSnapshotsUrlhttps://repository.apache.org/content/repositories/snapshots/distMgmtSnapshotsUrl
+distMgmtStagingIdapache.staging.https/distMgmtStagingId
+distMgmtStagingNameApache Release Distribution
Repository/distMgmtStagingName
+
distMgmtStagingUrlhttps://repository.apache.org/service/local/staging/deploy/maven2/distMgmtStagingUrl
project.build.sourceEncodingUTF-8/project.build.sourceEncoding
project.reporting.outputEncodingUTF-8/project.reporting.outputEncoding
@@ -1397,7 +1415,26 @@
/build
/profile
-
+profile
+ idsign/id
+ build
+plugins
+ plugin
+groupIdorg.apache.maven.plugins/groupId
+artifactIdmaven-gpg-plugin/artifactId
+executions
+ execution
+idsign-artifacts/id
+phaseverify/phase
+goals
+ goalsign/goal
+/goals
+ /execution
+/executions
+ /plugin
+/plugins
+ /build
+/profile
/profiles
[16/50] git commit: SLIDER-379. Modify pom version for the HBase package
SLIDER-379. Modify pom version for the HBase package Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/82cf1f01 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/82cf1f01 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/82cf1f01 Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 82cf1f010fbea58557003216d01341f832d3d6ac Parents: 4c10613 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sat Aug 30 10:11:56 2014 -0700 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sat Aug 30 10:12:05 2014 -0700 -- app-packages/hbase/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/82cf1f01/app-packages/hbase/pom.xml -- diff --git a/app-packages/hbase/pom.xml b/app-packages/hbase/pom.xml index 442b0f2..6f1da9c 100644 --- a/app-packages/hbase/pom.xml +++ b/app-packages/hbase/pom.xml @@ -19,7 +19,7 @@ parent groupIdorg.apache.slider/groupId artifactIdslider/artifactId -version0.41.0-incubating-SNAPSHOT/version +version0.51.0-incubating-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent modelVersion4.0.0/modelVersion
git commit: SLIDER-397 taskkill /f
Repository: incubator-slider
Updated Branches:
refs/heads/feature/SLIDER-397_windows_test_kill()_command_failing 132d9f288
- 27f0a6252
SLIDER-397 taskkill /f
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/27f0a625
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/27f0a625
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/27f0a625
Branch: refs/heads/feature/SLIDER-397_windows_test_kill()_command_failing
Commit: 27f0a625265de8e0d818917c94b2f0d5385b9044
Parents: 132d9f2
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 20:39:05 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 20:39:05 2014 +0100
--
.../test/groovy/org/apache/slider/test/SliderTestUtils.groovy| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/27f0a625/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
b/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
index 89f1a34..5086248 100644
--- a/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
@@ -747,7 +747,7 @@ class SliderTestUtils extends Assert {
def commandString
if (!Shell.WINDOWS) {
- GString killCommand = jps -l| grep ${grepString} | awk '{print \$1}' |
xargs kill $signal
+ GString killCommand = jps -l| grep ${grepString} | awk '{print \$1}' |
xargs kill /f $signal
log.info(Command command = $killCommand )
commandString = [bash, -c, killCommand]
@@ -755,7 +755,7 @@ class SliderTestUtils extends Assert {
/*
jps -l | grep String | awk {print $1} | xargs -n 1 taskkill /PID
*/
- GString killCommand = \jps -l | grep \${grepString}\ | gawk \{print
\$1}\ | xargs -n 1 taskkill /PID\
+ GString killCommand = \jps -l | grep \${grepString}\ | gawk \{print
\$1}\ | xargs -n 1 taskkill /f /PID
commandString = [CMD, /C, killCommand]
}
Process command = commandString.execute()
[24/50] git commit: Merge branch 'feature/SLIDER-327_AM_to_move_to_optimized, _queued_Node_review_and_refresh_action' into develop
Merge branch 'feature/SLIDER-327_AM_to_move_to_optimized,_queued_Node_review_and_refresh_action' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e7dd217a Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e7dd217a Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e7dd217a Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: e7dd217a324c205826bea4ac040b808c4ad72d61 Parents: b525287 88b665d Author: Steve Loughran ste...@apache.org Authored: Tue Sep 2 21:02:29 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Tue Sep 2 21:02:29 2014 +0100 -- .../org/apache/slider/client/SliderClient.java | 9 +-- .../server/appmaster/RoleLaunchService.java | 8 +-- .../server/appmaster/SliderAppMaster.java | 66 ++-- .../server/appmaster/actions/ActionHalt.java| 6 +- .../appmaster/actions/ActionKillContainer.java | 20 +- .../appmaster/actions/ActionStartContainer.java | 8 ++- .../appmaster/actions/ActionStopSlider.java | 6 +- .../server/appmaster/actions/AsyncAction.java | 47 -- .../actions/ProviderReportedContainerLoss.java | 13 ++-- .../actions/ProviderStartupCompleted.java | 4 -- .../server/appmaster/actions/QueueAccess.java | 5 ++ .../server/appmaster/actions/QueueService.java | 11 +++- .../actions/RegisterComponentInstance.java | 5 ++ .../actions/ReviewAndFlexApplicationSize.java | 43 + .../actions/UnregisterComponentInstance.java| 7 ++- .../server/appmaster/monkey/ChaosEntry.java | 2 +- .../TestStandaloneAMMonkeyRestart.groovy| 8 +-- .../providers/accumulo/AccumuloTestBase.groovy | 4 +- .../minicluster/HBaseMiniClusterTestBase.groovy | 1 - .../flexing/TestClusterFlex1To1.groovy | 2 +- 20 files changed, 176 insertions(+), 99 deletions(-) --
[23/50] git commit: SLIDER-327 AM to move to optimized, queued Node review and refresh action
SLIDER-327 AM to move to optimized, queued Node review and refresh action
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/88b665db
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/88b665db
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/88b665db
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 88b665db938325bf2c221614f7ffadefe71be632
Parents: 82cf1f0
Author: Steve Loughran ste...@apache.org
Authored: Tue Sep 2 21:01:50 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Tue Sep 2 21:01:50 2014 +0100
--
.../org/apache/slider/client/SliderClient.java | 9 +--
.../server/appmaster/RoleLaunchService.java | 8 +--
.../server/appmaster/SliderAppMaster.java | 66 ++--
.../server/appmaster/actions/ActionHalt.java| 6 +-
.../appmaster/actions/ActionKillContainer.java | 20 +-
.../appmaster/actions/ActionStartContainer.java | 8 ++-
.../appmaster/actions/ActionStopSlider.java | 6 +-
.../server/appmaster/actions/AsyncAction.java | 47 --
.../actions/ProviderReportedContainerLoss.java | 13 ++--
.../actions/ProviderStartupCompleted.java | 4 --
.../server/appmaster/actions/QueueAccess.java | 5 ++
.../server/appmaster/actions/QueueService.java | 11 +++-
.../actions/RegisterComponentInstance.java | 5 ++
.../actions/ReviewAndFlexApplicationSize.java | 43 +
.../actions/UnregisterComponentInstance.java| 7 ++-
.../server/appmaster/monkey/ChaosEntry.java | 2 +-
.../TestStandaloneAMMonkeyRestart.groovy| 8 +--
.../providers/accumulo/AccumuloTestBase.groovy | 4 +-
.../minicluster/HBaseMiniClusterTestBase.groovy | 1 -
.../flexing/TestClusterFlex1To1.groovy | 2 +-
20 files changed, 176 insertions(+), 99 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/88b665db/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index f0da136..f6d761e 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -2002,12 +2002,9 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
log.info(Flexing running cluster);
SliderClusterProtocol appMaster = connect(instance);
SliderClusterOperations clusterOps = new
SliderClusterOperations(appMaster);
- if (clusterOps.flex(instanceDefinition.getResources())) {
-log.info(Cluster size updated);
-exitCode = EXIT_SUCCESS;
- } else {
-log.info(Requested size is the same as current size: no change);
- }
+ clusterOps.flex(instanceDefinition.getResources());
+ log.info(application instance size updated);
+ exitCode = EXIT_SUCCESS;
} else {
log.info(No running instance to update);
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/88b665db/slider-core/src/main/java/org/apache/slider/server/appmaster/RoleLaunchService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/RoleLaunchService.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/RoleLaunchService.java
index e8b6802..37824c8 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/RoleLaunchService.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/RoleLaunchService.java
@@ -29,7 +29,6 @@ import org.apache.slider.core.launch.ContainerLauncher;
import org.apache.slider.providers.ProviderRole;
import org.apache.slider.providers.ProviderService;
import org.apache.slider.server.appmaster.actions.ActionStartContainer;
-import org.apache.slider.server.appmaster.actions.AsyncAction;
import org.apache.slider.server.appmaster.actions.QueueAccess;
import org.apache.slider.server.appmaster.state.RoleInstance;
import org.apache.slider.server.appmaster.state.RoleStatus;
@@ -39,9 +38,9 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;
-import java.util.Queue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
/**
* A service for launching containers
@@ -215,9 +214,8 @@ public class RoleLaunchService
instance.roleId = role.id;
instance.environment = envDescription;
actionQueue.put(new ActionStartContainer(starting + containerRole,
-
[34/50] git commit: SLIDER-377 switch to branch 2.6 to see if failures arise
SLIDER-377 switch to branch 2.6 to see if failures arise Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/877a0160 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/877a0160 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/877a0160 Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 877a01609748546a021d5477b7cd52f7f1536c65 Parents: 93c3c77 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 14:11:29 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 14:11:29 2014 +0100 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/877a0160/pom.xml -- diff --git a/pom.xml b/pom.xml index 454918c..8292073 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,7 @@ !-- core artifacts -- -hadoop.version3.0.0-SNAPSHOT/hadoop.version +hadoop.version2.6.0-SNAPSHOT/hadoop.version hbase.version0.98.4-hadoop2/hbase.version accumulo.version1.6.1-SNAPSHOT/accumulo.version
[18/50] git commit: Revert SLIDER-187. Use start/stop instead of freeze/thaw
Revert SLIDER-187. Use start/stop instead of freeze/thaw This reverts commit 0350273fd96579c1179a1f74a7d144e525f71572. Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/33ccc65a Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/33ccc65a Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/33ccc65a Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests Commit: 33ccc65a512f261c81c2412ebe9cd07a280e7326 Parents: be548b8 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Fri Aug 29 11:58:53 2014 -0700 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sat Aug 30 10:12:05 2014 -0700 -- README.md | 4 +- slider-assembly/src/main/bash/README.md | 2 +- slider-assembly/src/main/bash/slider_destroy| 2 +- .../apache/slider/api/ClusterDescription.java | 2 +- .../org/apache/slider/client/SliderClient.java | 20 - .../common/SliderXMLConfKeysForTesting.java | 8 ++-- .../slider/common/params/ActionFreezeArgs.java | 10 ++--- .../slider/common/params/ActionThawArgs.java| 6 +-- .../apache/slider/common/params/ClientArgs.java | 4 +- .../common/params/LaunchArgsAccessor.java | 2 +- .../slider/common/params/SliderActions.java | 14 +++--- .../server/appmaster/state/NodeEntry.java | 2 +- .../server/appmaster/state/RoleHistory.java | 14 +++--- .../agent/actions/TestActionExists.groovy | 2 +- .../agent/freezethaw/TestFreezeCommands.groovy | 20 - .../TestFreezeThawMasterlessAM.groovy | 8 ++-- .../freezethaw/TestFreezeUnknownCluster.groovy | 4 +- .../standalone/TestBuildStandaloneAM.groovy | 8 ++-- .../standalone/TestStandaloneAMDestroy.groovy | 8 ++-- .../standalone/TestStandaloneRegistryAM.groovy | 2 +- .../slider/client/TestClientBadArgs.groovy | 2 +- .../slider/client/TestCommonArgParsing.groovy | 12 ++--- .../model/history/TestRoleHistoryRW.groovy | 2 +- .../slider/test/YarnMiniClusterTestBase.groovy | 20 ++--- .../src/test/resources/example-slider-test.xml | 8 ++-- .../framework/AgentCommandTestBase.groovy | 2 +- .../funtest/framework/CommandTestBase.groovy| 46 ++-- .../commands/UnknownClusterOperationsIT.groovy | 9 ++-- .../lifecycle/AgentClusterLifecycleIT.groovy| 42 +- .../src/test/manual/python/SliderTester.py | 4 +- .../FunctionalAccumuloM1T1GC1Mon1IT.groovy | 6 +-- .../accumulo/live/TestAccFreezeThaw.groovy | 2 +- .../funtest/HBaseClusterLifecycleIT.groovy | 38 .../TestFreezeThawClusterFromArchive.groovy | 2 +- .../build/TestBuildThawClusterM1W1.groovy | 10 ++--- ...reezeReconfigureThawLiveRegionService.groovy | 12 ++--- .../TestFreezeThawLiveRegionService.groovy | 4 +- .../minicluster/live/TestTwoLiveClusters.groovy | 4 +- src/test/clusters/sandbox/operations.md | 20 - src/test/clusters/ubuntu-secure/operations.md | 14 +++--- 40 files changed, 204 insertions(+), 197 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/33ccc65a/README.md -- diff --git a/README.md b/README.md index a25b83a..b91e477 100644 --- a/README.md +++ b/README.md @@ -22,9 +22,9 @@ Slider is a YARN application to deploy existing distributed applications on YARN monitor them and make them larger or smaller as desired -even while the cluster is running. -Clusters can be stopped and restarted later; the distribution +Clusters can be stopped, frozen and restarted, thawed later; the distribution of the deployed application across the YARN cluster is persisted -enabling -a best-effort placement close to the previous locations on a cluster start. +a best-effort placement close to the previous locations on a cluster thaw. Applications which remember the previous placement of data (such as HBase) can exhibit fast start-up times from this feature. http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/33ccc65a/slider-assembly/src/main/bash/README.md -- diff --git a/slider-assembly/src/main/bash/README.md b/slider-assembly/src/main/bash/README.md index d818aeb..a4b7b08 100644 --- a/slider-assembly/src/main/bash/README.md +++ b/slider-assembly/src/main/bash/README.md @@ -79,7 +79,7 @@ Tear Down - * slider_destroy will do the following - 1. Stop the slider application based on provided name + 1. Freeze the slider application based on provided name 2. Destory the slider application based on provided name * The following args are required
[14/50] git commit: SLIDER-378. Slider AM restart- Logfolders and componentinstancedata not getting published
SLIDER-378. Slider AM restart- Logfolders and componentinstancedata not getting
published
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/11f6f2b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/11f6f2b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/11f6f2b3
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 11f6f2b3738f8b8c05be0324eec375e270ce1b3c
Parents: ae4c076
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Fri Aug 29 23:42:21 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Fri Aug 29 23:42:21 2014 -0700
--
.../src/main/python/agent/Controller.py | 1 +
.../python/agent/CustomServiceOrchestrator.py | 10 +-
slider-agent/src/main/python/agent/Register.py | 5 +-
.../src/test/python/agent/TestRegistration.py | 5 +-
.../providers/agent/AgentProviderService.java | 134 +++
.../appmaster/web/rest/agent/Register.java | 13 ++
.../agent/TestAgentProviderService.java | 18 ++-
7 files changed, 123 insertions(+), 63 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/11f6f2b3/slider-agent/src/main/python/agent/Controller.py
--
diff --git a/slider-agent/src/main/python/agent/Controller.py
b/slider-agent/src/main/python/agent/Controller.py
index a3fb90d..11db21c 100644
--- a/slider-agent/src/main/python/agent/Controller.py
+++ b/slider-agent/src/main/python/agent/Controller.py
@@ -122,6 +122,7 @@ class Controller(threading.Thread):
self.componentActualState,
self.componentExpectedState,
self.actionQueue.customServiceOrchestrator.allocated_ports,
+ self.actionQueue.customServiceOrchestrator.log_folders,
id))
logger.info(Registering with the server at + self.registerUrl +
with data + pprint.pformat(data))
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/11f6f2b3/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
--
diff --git a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
index 15f1664..dd8e9b9 100644
--- a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
+++ b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
@@ -26,6 +26,7 @@ import sys
import socket
import posixpath
import platform
+import copy
from AgentConfig import AgentConfig
from AgentException import AgentException
from PythonExecutor import PythonExecutor
@@ -58,6 +59,7 @@ class CustomServiceOrchestrator():
self.public_fqdn = hostname.public_hostname()
self.stored_command = {}
self.allocated_ports = {}
+self.log_folders = {}
# Clean up old status command files if any
try:
os.unlink(self.status_commands_stdout)
@@ -133,15 +135,17 @@ class CustomServiceOrchestrator():
}
if Constants.EXIT_CODE in ret and ret[Constants.EXIT_CODE] == 0:
- ret[Constants.ALLOCATED_PORTS] = allocated_ports
- self.allocated_ports = allocated_ports
+ ret[Constants.ALLOCATED_PORTS] = copy.deepcopy(allocated_ports)
+ ## Generally all ports are allocated at once but just in case
+ self.allocated_ports.update(allocated_ports)
# Irrespective of the outcome report the folder paths
if command_name == 'INSTALL':
- ret[Constants.FOLDERS] = {
+ self.log_folders = {
Constants.AGENT_LOG_ROOT: self.config.getLogPath(),
Constants.AGENT_WORK_ROOT: self.config.getWorkRootPath()
}
+ ret[Constants.FOLDERS] = copy.deepcopy(self.log_folders)
return ret
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/11f6f2b3/slider-agent/src/main/python/agent/Register.py
--
diff --git a/slider-agent/src/main/python/agent/Register.py
b/slider-agent/src/main/python/agent/Register.py
index b59154f..c8246c7 100644
--- a/slider-agent/src/main/python/agent/Register.py
+++ b/slider-agent/src/main/python/agent/Register.py
@@ -29,7 +29,7 @@ class Register:
def __init__(self, config):
self.config = config
- def build(self, actualState, expectedState, allocated_ports, id='-1'):
+ def build(self, actualState, expectedState, allocated_ports, log_folders,
id='-1'):
timestamp = int(time.time() * 1000)
version = self.read_agent_version()
@@ -41,7 +41,8 @@ class Register:
'agentVersion': version,
'actualState': actualState,
'expectedState': expectedState,
-
[37/50] git commit: SLIDER-394 slider client and AM services to fail-fast if the hadoop dependencies are missing
SLIDER-394 slider client and AM services to fail-fast if the hadoop
dependencies are missing
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0f67b104
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0f67b104
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0f67b104
Branch: refs/heads/feature/SLIDER-280_Restart_AM_fun_tests
Commit: 0f67b10424cb807aebb3d2b15c3b24149f4ae70a
Parents: 29a40f5
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 16:28:20 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 16:28:20 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 555 +--
.../services/utility/EndOfServiceWaiter.java| 84 +++
.../services/workflow/ForkedProcessService.java | 21 +-
.../services/workflow/LongLivedProcess.java | 38 +-
.../tools/TestExecutionEnvironment.groovy | 42 ++
.../common/tools/TestWindowsSupport.groovy | 4 +-
.../services/workflow/EndOfServiceWaiter.java | 56 --
.../TestWorkflowForkedProcessService.java | 4 +-
8 files changed, 570 insertions(+), 234 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0f67b104/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index eb214db..44fe4fd 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -56,7 +56,10 @@ import org.apache.slider.core.exceptions.ErrorStrings;
import org.apache.slider.core.exceptions.MissingArgException;
import org.apache.slider.core.exceptions.SliderException;
import org.apache.slider.core.launch.ClasspathConstructor;
+import org.apache.slider.core.main.LauncherExitCodes;
+import org.apache.slider.server.services.utility.EndOfServiceWaiter;
import org.apache.slider.server.services.utility.PatternValidator;
+import org.apache.slider.server.services.workflow.ForkedProcessService;
import org.apache.zookeeper.server.util.KerberosUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -67,6 +70,7 @@ import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
+import java.io.InterruptedIOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.net.InetSocketAddress;
@@ -88,6 +92,7 @@ import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
import java.util.TreeSet;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
/**
@@ -102,12 +107,24 @@ public final class SliderUtils {
* turned on (prevents re-entrancy)
*/
private static final AtomicBoolean processSecurityAlreadyInitialized =
-new AtomicBoolean(false);
+ new AtomicBoolean(false);
public static final String JAVA_SECURITY_KRB5_REALM =
java.security.krb5.realm;
public static final String JAVA_SECURITY_KRB5_KDC = java.security.krb5.kdc;
+
+ /**
+ * Winutils
+ */
public static final String WINUTILS = WINUTILS.EXE;
+ /**
+ * name of openssl program
+ */
+ public static final String OPENSSL = openssl;
+ /**
+ * name of python program
+ */
+ public static final String PYTHON = python;
private SliderUtils() {
}
@@ -131,7 +148,8 @@ public final class SliderUtils {
* @param msg the message to be shown in exception
*/
@SuppressWarnings(ResultOfMethodCallIgnored)
- private static void validateNumber(String num, String msg) throws
BadConfigException {
+ private static void validateNumber(String num, String msg) throws
+ BadConfigException {
try {
Integer.parseInt(num);
} catch (NumberFormatException nfe) {
@@ -145,15 +163,16 @@ public final class SliderUtils {
* @param heapsize
* @return heapsize in MB
*/
- public static String translateTrailingHeapUnit(String heapsize) throws
BadConfigException {
+ public static String translateTrailingHeapUnit(String heapsize) throws
+ BadConfigException {
String errMsg = Bad heapsize: ;
if (heapsize.endsWith(m) || heapsize.endsWith(M)) {
- String num = heapsize.substring(0, heapsize.length()-1);
+ String num = heapsize.substring(0, heapsize.length() - 1);
validateNumber(num, errMsg);
return num;
}
if (heapsize.endsWith(g) || heapsize.endsWith(G)) {
- String num = heapsize.substring(0, heapsize.length()-1)+000;
+ String num = heapsize.substring(0,
[4/4] git commit: Merge branch 'feature/SLIDER-397_windows_test_kill()_command_failing' into develop
Merge branch 'feature/SLIDER-397_windows_test_kill()_command_failing' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e2a80326 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e2a80326 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e2a80326 Branch: refs/heads/develop Commit: e2a80326a7f7ddc6f2bc8b00dbaa5265726a6bb0 Parents: 930920d ee3ad77 Author: Steve Loughran ste...@apache.org Authored: Thu Sep 4 21:02:30 2014 +0100 Committer: Steve Loughran ste...@apache.org Committed: Thu Sep 4 21:02:30 2014 +0100 -- .../apache/slider/agent/standalone/TestStandaloneAMKill.groovy | 2 +- .../test/groovy/org/apache/slider/test/SliderTestUtils.groovy| 4 ++-- .../groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) --
[1/4] git commit: SLIDER-397 windows kill() command failing in TestStandaloneAMKill
Repository: incubator-slider
Updated Branches:
refs/heads/develop 930920d24 - e2a80326a
SLIDER-397 windows kill() command failing in TestStandaloneAMKill
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/132d9f28
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/132d9f28
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/132d9f28
Branch: refs/heads/develop
Commit: 132d9f288a9d91790409bd4482731eb27cf58bba
Parents: 930920d
Author: Steve Loughran ste...@apache.org
Authored: Thu Sep 4 20:31:29 2014 +0100
Committer: Steve Loughran ste...@apache.org
Committed: Thu Sep 4 20:31:29 2014 +0100
--
.../apache/slider/agent/standalone/TestStandaloneAMKill.groovy | 2 +-
.../groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/132d9f28/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
index 75f9a2c..7552394 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMKill.groovy
@@ -58,7 +58,7 @@ class TestStandaloneAMKill extends AgentMiniClusterTestBase {
describe(listing Java processes)
lsJavaProcesses();
describe(killing AM)
-killAM(SIGTERM);
+assert 0 == killAM(SIGTERM);
waitWhileClusterLive(sliderClient);
//give yarn some time to notice
sleep(1)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/132d9f28/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
index 7ffa64a..97cc853 100644
---
a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
@@ -318,10 +318,10 @@ public abstract class YarnMiniClusterTestBase extends
ServiceLauncherBaseTest {
/**
- * Kill all Slider Services. That i
+ * Kill all Slider Services.
* @param signal
*/
- public void killAM(int signal) {
+ public int killAM(int signal) {
killJavaProcesses(SliderAppMaster.SERVICE_CLASSNAME_SHORT, signal)
}
Git Push Summary
Repository: incubator-slider Updated Branches: refs/heads/feature/SLIDER-397_windows_test_kill()_command_failing [deleted] ee3ad77a8
svn commit: r1622558 - in /incubator/slider/site/trunk/content/developing: building.md windows.md
Author: stevel Date: Thu Sep 4 20:37:02 2014 New Revision: 1622558 URL: http://svn.apache.org/r1622558 Log: SLIDER-86 update docs on windows Modified: incubator/slider/site/trunk/content/developing/building.md incubator/slider/site/trunk/content/developing/windows.md Modified: incubator/slider/site/trunk/content/developing/building.md URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/building.md?rev=1622558r1=1622557r2=1622558view=diff == --- incubator/slider/site/trunk/content/developing/building.md (original) +++ incubator/slider/site/trunk/content/developing/building.md Thu Sep 4 20:37:02 2014 @@ -28,9 +28,9 @@ The network on the development system mu of the local host working. Tests will fail without this. For maven builds to work, remote network access is often a pre-requisite. -### Java 7 +### Java -Slider is built on Java 7 -please have a JDK for Java 7 or 8 set up +Slider is built on Java 6 or later ### Python @@ -60,7 +60,10 @@ This is absolutely critical to prevent J mvn clean install -Slider code base includes unit tests and functional tests. By default functional tests are not run as these tests are developed to be run against live hadoop clusters and require some manual setup. You can run them based as described at [functional test](/developing/functional_tests.html). +Slider code base includes unit tests and functional tests. +By default functional tests are not run as these tests are developed to be run +against live hadoop clusters and require some manual setup. +You can run them based as described at [functional test](/developing/functional_tests.html). ### Create Slider Package @@ -68,7 +71,9 @@ Slider code base includes unit tests and mvn clean site:site site:stage package -DskipTests -***At this point you are ready to use the Slider toolset. The build instructions below are optional and are proven to be useful when debugging deep into the hadoop code base.*** +***At this point you are ready to use the Slider toolset. +The build instructions below are optional and are proven to +be useful when debugging deep into the hadoop code base.*** @@ -85,25 +90,24 @@ to have a local version of Hadoop -so th Hadoop as well in Slider. -To build and install locally, check out apache svn/github, branch `release-2.4.0`, +To build and install locally, check out apache svn/github, branch `branch-2`, and create a branch off that tag git clone git://git.apache.org/hadoop-common.git cd hadoop-common -git remote rename origin apache -git fetch --tags apache -git checkout release-2.4.0 -- -git checkout -b release-2.4.0 +git fetch --tags origin +git checkout -b origin/branch-2 + +git checkout -b branch-2 +export HADOOP_VERSION=2.6.0-SNAPSHOT -For the scripts below, set the `HADOOP_VERSION` variable to the version +(to build against a release, check out that specific release and create +a branch off it:) + +git checkout release-2.6.0 -- +git checkout -b release-2.6.0 -export HADOOP_VERSION=2.4.0 - -or, for building against a pre-release version of Hadoop 2.4 - -git checkout branch-2 -export HADOOP_VERSION=2.4.0-SNAPSHOT To build and install it locally, skipping the tests: @@ -117,13 +121,14 @@ To make a tarball for use in test runs: # on linux mvn clean package -Pdist -Pnative -Dtar -DskipTests -Dmaven.javadoc.skip=true -Then expand this +Then clean expand this pushd hadoop-dist/target/ -gunzip hadoop-$HADOOP_VERSION.tar.gz -tar -xvf hadoop-$HADOOP_VERSION.tar +tar -xvzf hadoop-$HADOOP_VERSION.tar.gz popd + + This creates an expanded version of Hadoop. You can now actually run Hadoop from this directory. Do note that unless you have the native code built for your target platform, Hadoop will be slower. @@ -393,7 +398,8 @@ either stream for logging Example: - target/TestLiveRegionService/TestLiveRegionService-logDir-nm-1_0/application_1376095770244_0001/container_1376095770244_0001_01_01/out.txt +target/TestLiveRegionService/TestLiveRegionService-logDir-nm-1_0/ + application_1376095770244_0001/container_1376095770244_0001_01_01/out.txt 1. The actual test log from JUnit itself goes to the console and into `target/surefire/`; this shows the events happening in the YARN services as well Modified: incubator/slider/site/trunk/content/developing/windows.md URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/windows.md?rev=1622558r1=1622557r2=1622558view=diff == --- incubator/slider/site/trunk/content/developing/windows.md (original) +++ incubator/slider/site/trunk/content/developing/windows.md Thu Sep 4 20:37:02 2014 @@ -31,20 +31,39 @@
git commit: SLIDER-400 added zero port properties to sample appConfigs
Repository: incubator-slider
Updated Branches:
refs/heads/develop e2a80326a - 168a7af02
SLIDER-400 added zero port properties to sample appConfigs
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/168a7af0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/168a7af0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/168a7af0
Branch: refs/heads/develop
Commit: 168a7af0251abb7d4b1c8017360fc21531ca83b7
Parents: e2a8032
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Thu Sep 4 20:22:40 2014 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Thu Sep 4 20:22:40 2014 -0700
--
app-packages/accumulo/appConfig.json | 2 ++
app-packages/accumulo/src/test/resources/appConfig_kerberos.json | 2 ++
.../accumulo/src/test/resources/appConfig_monitor_ssl.json | 2 ++
app-packages/accumulo/src/test/resources/appConfig_ssl.json| 2 ++
4 files changed, 8 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/168a7af0/app-packages/accumulo/appConfig.json
--
diff --git a/app-packages/accumulo/appConfig.json
b/app-packages/accumulo/appConfig.json
index 2c0c2d5..8828a64 100644
--- a/app-packages/accumulo/appConfig.json
+++ b/app-packages/accumulo/appConfig.json
@@ -40,6 +40,8 @@
site.accumulo-site.gc.port.client: 0,
site.accumulo-site.monitor.port.client:
${ACCUMULO_MONITOR.ALLOCATED_PORT},
site.accumulo-site.monitor.port.log4j: 0,
+site.accumulo-site.master.replication.coordinator.port: 0,
+site.accumulo-site.replication.receipt.service.port: 0,
site.accumulo-site.general.classpaths:
$ACCUMULO_HOME/lib/accumulo-server.jar,\n$ACCUMULO_HOME/lib/accumulo-core.jar,\n$ACCUMULO_HOME/lib/accumulo-start.jar,\n$ACCUMULO_HOME/lib/accumulo-fate.jar,\n$ACCUMULO_HOME/lib/accumulo-proxy.jar,\n$ACCUMULO_HOME/lib/[^.].*.jar,\n$ZOOKEEPER_HOME/zookeeper[^.].*.jar,\n$HADOOP_CONF_DIR,\n$HADOOP_PREFIX/[^.].*.jar,\n$HADOOP_PREFIX/lib/[^.].*.jar,\n$HADOOP_PREFIX/share/hadoop/common/.*.jar,\n$HADOOP_PREFIX/share/hadoop/common/lib/.*.jar,\n$HADOOP_PREFIX/share/hadoop/hdfs/.*.jar,\n$HADOOP_PREFIX/share/hadoop/mapreduce/.*.jar,\n$HADOOP_PREFIX/share/hadoop/yarn/.*.jar,\n/usr/lib/hadoop/.*.jar,\n/usr/lib/hadoop/lib/.*.jar,\n/usr/lib/hadoop-hdfs/.*.jar,\n/usr/lib/hadoop-mapreduce/.*.jar,\n/usr/lib/hadoop-yarn/.*.jar,
},
credentials: {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/168a7af0/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
--
diff --git a/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
b/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
index c9684d4..b1bff37 100644
--- a/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
+++ b/app-packages/accumulo/src/test/resources/appConfig_kerberos.json
@@ -42,6 +42,8 @@
site.accumulo-site.gc.port.client: 0,
site.accumulo-site.monitor.port.client:
${ACCUMULO_MONITOR.ALLOCATED_PORT},
site.accumulo-site.monitor.port.log4j: 0,
+site.accumulo-site.master.replication.coordinator.port: 0,
+site.accumulo-site.replication.receipt.service.port: 0,
site.accumulo-site.general.classpaths:
$ACCUMULO_HOME/lib/accumulo-server.jar,\n$ACCUMULO_HOME/lib/accumulo-core.jar,\n$ACCUMULO_HOME/lib/accumulo-start.jar,\n$ACCUMULO_HOME/lib/accumulo-fate.jar,\n$ACCUMULO_HOME/lib/accumulo-proxy.jar,\n$ACCUMULO_HOME/lib/[^.].*.jar,\n$ZOOKEEPER_HOME/zookeeper[^.].*.jar,\n$HADOOP_CONF_DIR,\n$HADOOP_PREFIX/[^.].*.jar,\n$HADOOP_PREFIX/lib/[^.].*.jar,\n$HADOOP_PREFIX/share/hadoop/common/.*.jar,\n$HADOOP_PREFIX/share/hadoop/common/lib/.*.jar,\n$HADOOP_PREFIX/share/hadoop/hdfs/.*.jar,\n$HADOOP_PREFIX/share/hadoop/mapreduce/.*.jar,\n$HADOOP_PREFIX/share/hadoop/yarn/.*.jar,\n/usr/lib/hadoop/.*.jar,\n/usr/lib/hadoop/lib/.*.jar,\n/usr/lib/hadoop-hdfs/.*.jar,\n/usr/lib/hadoop-mapreduce/.*.jar,\n/usr/lib/hadoop-yarn/.*.jar,
},
credentials: {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/168a7af0/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
--
diff --git
a/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
b/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
index 45e6e19..61a9bb1 100644
--- a/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
+++ b/app-packages/accumulo/src/test/resources/appConfig_monitor_ssl.json
@@ -40,6 +40,8 @@
site.accumulo-site.gc.port.client: 0,
site.accumulo-site.monitor.port.client:
git commit: SLIDER-399. Agent command doesn't consider the case where LOG_DIRS contains multiple directories
Repository: incubator-slider
Updated Branches:
refs/heads/feature/auto_generate_storm_pkg 02faa944d - 4c808204d
SLIDER-399. Agent command doesn't consider the case where LOG_DIRS contains
multiple directories
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/4c808204
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/4c808204
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/4c808204
Branch: refs/heads/feature/auto_generate_storm_pkg
Commit: 4c808204d749f1504b768b12a98bf64f583e49d3
Parents: 02faa94
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Sep 4 21:50:52 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Sep 4 21:50:52 2014 -0700
--
.../main/java/org/apache/slider/providers/agent/AgentKeys.java| 1 +
.../org/apache/slider/providers/agent/AgentProviderService.java | 3 +--
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4c808204/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
index 1622367..1da99b0 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
@@ -91,6 +91,7 @@ public interface AgentKeys {
String CREATE_DEF_ZK_NODE = create.default.zookeeper.node;
String HEARTBEAT_MONITOR_INTERVAL = heartbeat.monitor.interval;
String AGENT_INSTANCE_DEBUG_DATA = agent.instance.debug.data;
+ String AGENT_OUT_FILE = agent.out;
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4c808204/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index d188d49..02ef625 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -333,8 +333,7 @@ public class AgentProviderService extends
AbstractProviderService implements
operation.add(debugCmd);
}
-String outfile = new File(logDir, agent.out).toString();
-operation.add( + outfile + 21);
+operation.add( + ApplicationConstants.LOG_DIR_EXPANSION_VAR + / +
AgentKeys.AGENT_OUT_FILE + 21);
launcher.addCommand(operation.build());
[1/5] git commit: SLIDER-181. Storm package updates
Repository: incubator-slider
Updated Branches:
refs/heads/develop 168a7af02 - 56cd86672
SLIDER-181. Storm package updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e1129b06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e1129b06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e1129b06
Branch: refs/heads/develop
Commit: e1129b0609621356b47324dcbcabd612fd275be5
Parents: 201686e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Sep 3 13:15:20 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Sep 3 13:15:20 2014 -0700
--
app-packages/storm/README.txt | 19 ++---
app-packages/storm/appConfig.json | 10 +--
app-packages/storm/metainfo.xml | 2 +-
...pache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE | 16
app-packages/storm/pom.xml | 90
app-packages/storm/src/assembly/storm.xml | 68 +++
6 files changed, 172 insertions(+), 33 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e1129b06/app-packages/storm/README.txt
--
diff --git a/app-packages/storm/README.txt b/app-packages/storm/README.txt
index d4df85f..49adfce 100644
--- a/app-packages/storm/README.txt
+++ b/app-packages/storm/README.txt
@@ -17,19 +17,16 @@
How to create a Slider app package for Storm?
-To create the app package you will need the Storm tarball copied to a specific
location.
-Various configurations provided in this sample are customized for
apache-storm-0.9.3.2.2.0.0-578.tar.gz.
-So if you use a different version you may need to edit a few config values.
+To create the app package you will need the Storm tarball and invoke mvn
command
+with appropriate parameters.
-Replace the placeholder tarball for Storm.
- cp ~/Downloads/apache-storm-0.9.3.2.2.0.0-578.tar.gz package/files/
- rm package/files/apache-storm-0.9.3.2.2.0.0-578.tar.gz.REPLACE
+Command:
+mvn clean package -Pstorm-app-package -Dpkg.version=version
+ -Dpkg.name=file name of app tarball -Dpkg.src=folder location where the
pkg is available
-Create a zip package at the root of the package (slider
enlistment/app-packages/storm/)
- zip -r apache-storm-0.9.3.zip .
-
-Verify the content using
- unzip -l $@ apache-storm-0.9.3.zip
+Example:
+mvn clean package -Pstorm-app-package -Dpkg.version=0.9.3.2.2.0.0-578
+ -Dpkg.name=apache-storm-0.9.3.2.2.0.0-578.tar.gz
-Dpkg.src=/Users/user1/Downloads
While appConfig.json and resources.json are not required for the package they
work
well as the default configuration for Slider apps. So its advisable that when
you
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e1129b06/app-packages/storm/appConfig.json
--
diff --git a/app-packages/storm/appConfig.json
b/app-packages/storm/appConfig.json
index b5fe436..df28e8f 100644
--- a/app-packages/storm/appConfig.json
+++ b/app-packages/storm/appConfig.json
@@ -3,13 +3,13 @@
metadata: {
},
global: {
-application.def: apache-storm-0.9.3.zip,
+application.def: slider-storm-app-package-${pkg.version}.zip,
java_home: /usr/jdk64/jdk1.7.0_45,
create.default.zookeeper.node: true,
system_configs: core-site,
site.global.app_user: yarn,
-site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578,
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version},
site.global.user_group: hadoop,
site.global.ganglia_server_host: ${NN_HOST},
site.global.ganglia_server_id: Application2,
@@ -22,8 +22,8 @@
site.storm-site.storm.local.dir: ${AGENT_WORK_ROOT}/app/tmp/storm,
site.storm-site.transactional.zookeeper.root: /transactional,
site.storm-site.storm.zookeeper.port: 2181,
-site.storm-site.nimbus.childopts: -Xmx1024m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM,
-site.storm-site.worker.childopts: -Xmx768m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.2.2.0.0-578/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM,
+site.storm-site.nimbus.childopts: -Xmx1024m
[2/5] git commit: Merge branch 'develop' into feature/auto_generate_storm_pkg
Merge branch 'develop' into feature/auto_generate_storm_pkg Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/26b4153e Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/26b4153e Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/26b4153e Branch: refs/heads/develop Commit: 26b4153ebf070521fb5030d57846f4ccd7f45cb0 Parents: e1129b0 38c816e Author: Sumit Mohanty smoha...@hortonworks.com Authored: Wed Sep 3 13:46:44 2014 -0700 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Wed Sep 3 13:46:44 2014 -0700 -- .../main/java/org/apache/slider/api/InternalKeys.java | 11 +++ .../main/java/org/apache/slider/api/ResourceKeys.java | 13 - .../apache/slider/server/appmaster/state/AppState.java | 9 + 3 files changed, 16 insertions(+), 17 deletions(-) --
[3/5] git commit: SLIDER-391. appConfig.json should allow reference to other config properties and resolve them automatically
SLIDER-391. appConfig.json should allow reference to other config properties
and resolve them automatically
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/02faa944
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/02faa944
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/02faa944
Branch: refs/heads/develop
Commit: 02faa944da56cd6252b48afbdb6ee427291f1fe0
Parents: 26b4153
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Sep 3 18:17:13 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Sep 3 18:17:13 2014 -0700
--
app-packages/hbase/appConfig.json | 8 +++--
app-packages/storm/appConfig.json | 7 ++--
.../providers/agent/AgentProviderService.java | 35
.../agent/TestAgentProviderService.java | 29
4 files changed, 66 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/02faa944/app-packages/hbase/appConfig.json
--
diff --git a/app-packages/hbase/appConfig.json
b/app-packages/hbase/appConfig.json
index 7df0dc3..07d1b4e 100644
--- a/app-packages/hbase/appConfig.json
+++ b/app-packages/hbase/appConfig.json
@@ -11,14 +11,16 @@
site.global.app_user: yarn,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/hbase-${hbase.version},
+site.global.ganglia_server_host: ${NN_HOST},
+site.global.ganglia_server_port: 8667,
+site.global.ganglia_server_id: Application1,
+site.global.ganglia_enabled:true,
+
site.global.hbase_instance_name: instancename,
site.global.hbase_root_password: secret,
site.global.user_group: hadoop,
site.global.security_enabled: false,
site.global.monitor_protocol: http,
-site.global.ganglia_server_host: ${NN_HOST},
-site.global.ganglia_server_port: 8667,
-site.global.ganglia_server_id: Application1,
site.global.hbase_thrift_port: ${HBASE_THRIFT.ALLOCATED_PORT},
site.global.hbase_thrift2_port: ${HBASE_THRIFT2.ALLOCATED_PORT},
site.global.hbase_rest_port: ${HBASE_REST.ALLOCATED_PORT},
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/02faa944/app-packages/storm/appConfig.json
--
diff --git a/app-packages/storm/appConfig.json
b/app-packages/storm/appConfig.json
index df28e8f..ac9bd49 100644
--- a/app-packages/storm/appConfig.json
+++ b/app-packages/storm/appConfig.json
@@ -11,6 +11,7 @@
site.global.app_user: yarn,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version},
site.global.user_group: hadoop,
+
site.global.ganglia_server_host: ${NN_HOST},
site.global.ganglia_server_id: Application2,
site.global.ganglia_enabled:true,
@@ -22,8 +23,8 @@
site.storm-site.storm.local.dir: ${AGENT_WORK_ROOT}/app/tmp/storm,
site.storm-site.transactional.zookeeper.root: /transactional,
site.storm-site.storm.zookeeper.port: 2181,
-site.storm-site.nimbus.childopts: -Xmx1024m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM,
-site.storm-site.worker.childopts: -Xmx768m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM,
+site.storm-site.nimbus.childopts: -Xmx1024m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${site.global.ganglia_server_host},port=${site.global.ganglia_server_port},wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM,
+site.storm-site.worker.childopts: -Xmx768m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${site.global.ganglia_server_host},port=${site.global.ganglia_server_port},wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM,
site.storm-site.dev.zookeeper.path:
[4/5] git commit: SLIDER-399. Agent command doesn't consider the case where LOG_DIRS contains multiple directories
SLIDER-399. Agent command doesn't consider the case where LOG_DIRS contains
multiple directories
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/4c808204
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/4c808204
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/4c808204
Branch: refs/heads/develop
Commit: 4c808204d749f1504b768b12a98bf64f583e49d3
Parents: 02faa94
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Sep 4 21:50:52 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Sep 4 21:50:52 2014 -0700
--
.../main/java/org/apache/slider/providers/agent/AgentKeys.java| 1 +
.../org/apache/slider/providers/agent/AgentProviderService.java | 3 +--
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4c808204/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
index 1622367..1da99b0 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentKeys.java
@@ -91,6 +91,7 @@ public interface AgentKeys {
String CREATE_DEF_ZK_NODE = create.default.zookeeper.node;
String HEARTBEAT_MONITOR_INTERVAL = heartbeat.monitor.interval;
String AGENT_INSTANCE_DEBUG_DATA = agent.instance.debug.data;
+ String AGENT_OUT_FILE = agent.out;
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4c808204/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index d188d49..02ef625 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -333,8 +333,7 @@ public class AgentProviderService extends
AbstractProviderService implements
operation.add(debugCmd);
}
-String outfile = new File(logDir, agent.out).toString();
-operation.add( + outfile + 21);
+operation.add( + ApplicationConstants.LOG_DIR_EXPANSION_VAR + / +
AgentKeys.AGENT_OUT_FILE + 21);
launcher.addCommand(operation.build());
