svn commit: r1607095 - /incubator/slider/site/trunk/content/docs/security.md
Author: stevel Date: Tue Jul 1 14:25:30 2014 New Revision: 1607095 URL: http://svn.apache.org/r1607095 Log: SLIDER-192: use hadoop.security.authentication type as indicate whether to use security or not. Modified: incubator/slider/site/trunk/content/docs/security.md Modified: incubator/slider/site/trunk/content/docs/security.md URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/security.md?rev=1607095&r1=1607094&r2=1607095&view=diff == --- incubator/slider/site/trunk/content/docs/security.md (original) +++ incubator/slider/site/trunk/content/docs/security.md Tue Jul 1 14:25:30 2014 @@ -24,7 +24,7 @@ to deploy secure applications on a secur This document does not cover Kerberos, how to secure a Hadoop cluster, Kerberos command line tools or how Hadoop uses delegation tokens to delegate permissions -round a cluster. These are assumed, though some links to useful pages are +round a cluster. These are prerequisites &em;though some links to useful pages are listed at the bottom. @@ -72,20 +72,21 @@ Slider runs in secure clusters, but with * Slider application instance and HBase instance to remain functional and secure over an indefinite period of time. ### Initial Non-requirements -* secure audit trail of cluster operations. -* multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance. -* More than one HBase cluster in the YARN cluster belonging to a single user (irrespective of how they are started). +* Secure audit trail of cluster operations. +* Multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance. * Any way to revoke certificates/rights of running containers. ### Assumptions * Kerberos is running and that HDFS and YARN are running Kerberized. * LDAP cannot be assumed. -* Credentials needed for HBase can be pushed out into the local filesystems of +* Credentials needed for the application can be pushed out into the local filesystems of the of the worker nodes via some external mechanism (e.g. scp), and protected by the access permissions of the native filesystem. Any user with access to these credentials is considered to have been granted such rights. -* These credentials can outlive the duration of the HBase containers -* The user running HBase has the same identity as that of the HBase cluster. +* These credentials can outlive the duration of the application instances +* The user running the application has the same identity as that of the application. +* All application instances run by a single user can share the same machine-specific +kerberos identities. ## Design @@ -172,7 +173,7 @@ They can also be set on the Slider comma -S java.security.krb5.realm=MINICLUSTER -S java.security.krb5.kdc=hadoop-kdc -### Java Cryptography Exceptions +### Important: Java Cryptography Package When trying to talk to a secure, cluster you may see the message: @@ -184,6 +185,8 @@ needed to work with the keys that Kerber from Oracle (or other supplier of the JVM) and installed according to its accompanying instructions. + + ## Useful Links 1. [Adding Security to Apache Hadoop](http://hortonworks.com/wp-content/uploads/2011/10/security-design_withCover-1.pdf)
git commit: SLIDER-192: use hadoop.security.authentication type as indicate whether to use security or not.
Repository: incubator-slider
Updated Branches:
refs/heads/develop 31d22b888 -> 8fe71f067
SLIDER-192: use hadoop.security.authentication type as indicate whether to use
security or not.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8fe71f06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8fe71f06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8fe71f06
Branch: refs/heads/develop
Commit: 8fe71f06738321525501b7c13a50046bf85e43fb
Parents: 31d22b8
Author: Steve Loughran
Authored: Tue Jul 1 15:24:58 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 15:24:58 2014 +0100
--
.../java/org/apache/slider/client/SliderClient.java | 4 +++-
.../org/apache/slider/common/SliderXmlConfKeys.java | 4 ++--
.../org/apache/slider/common/tools/SliderUtils.java | 15 ---
3 files changed, 13 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8fe71f06/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index 95c120c..a337533 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -894,7 +894,9 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
if (clusterSecure) {
// if the cluster is secure, make sure that
// the relevant security settings go over
- addConfOptionToCLI(commandLine, config, KEY_SECURITY_ENABLED);
+/*
+ addConfOptionToCLI(commandLine, config, KEY_SECURITY);
+*/
addConfOptionToCLI(commandLine,
config,
DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY);
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8fe71f06/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java
b/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java
index 272ae6a..c7b8ea5 100644
--- a/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java
+++ b/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java
@@ -93,8 +93,8 @@ public interface SliderXmlConfKeys {
* Flag which is set to indicate that security should be enabled
* when talking to this cluster.
*/
- String KEY_SECURITY_ENABLED =
- CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION;
+ String KEY_SECURITY =
+ CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION;
/**
* queue name
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8fe71f06/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index b97ff63..5479b54 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -904,7 +904,8 @@ public final class SliderUtils {
* @return true if the slider client/service should be in secure mode
*/
public static boolean isHadoopClusterSecure(Configuration conf) {
-return conf.getBoolean(SliderXmlConfKeys.KEY_SECURITY_ENABLED, false);
+return SecurityUtil.getAuthenticationMethod(conf) !=
+ UserGroupInformation.AuthenticationMethod.SIMPLE;
}
/**
@@ -953,22 +954,22 @@ public final class SliderUtils {
conf.get(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION));
log.debug("hadoop.security.authorization={}",
conf.get(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION));
-SecurityUtil.setAuthenticationMethod(
-UserGroupInformation.AuthenticationMethod.KERBEROS, conf);
+/*SecurityUtil.setAuthenticationMethod(
+UserGroupInformation.AuthenticationMethod.KERBEROS, conf);*/
UserGroupInformation.setConfiguration(conf);
UserGroupInformation authUser = UserGroupInformation.getCurrentUser();
log.debug("Authenticating as " + authUser.toString());
log.debug("Login user is {}", UserGroupInformation.getLoginUser());
if (!UserGroupInformation.isSecurityEnabled()) {
throw new BadConfigException("Although secure mode is enabled," +
-
git commit: SLIDER-159: jenkins testEcho intermittently failing
Repository: incubator-slider
Updated Branches:
refs/heads/develop 8fe71f067 -> 9a54a0b8d
SLIDER-159: jenkins testEcho intermittently failing
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/9a54a0b8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/9a54a0b8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/9a54a0b8
Branch: refs/heads/develop
Commit: 9a54a0b8d3e6d5f5f706407dfe4ba432419b90d2
Parents: 8fe71f0
Author: Steve Loughran
Authored: Tue Jul 1 16:19:58 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 16:19:58 2014 +0100
--
.../services/workflow/ForkedProcessService.java | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9a54a0b8/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java
b/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java
index 7e73005..ee68aa4 100644
---
a/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java
+++
b/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java
@@ -281,12 +281,15 @@ public class ForkedProcessService extends
AbstractWorkflowExecutorService implem
}
long start = System.currentTimeMillis();
while (System.currentTimeMillis() - start <= duration) {
- if (finalOutput && process.isFinalOutputProcessed()) {
-//end of stream, break
-break;
- }
- if (!process.isRecentOutputEmpty()) {
+ boolean finished;
+ if (finalOutput) {
+// final flag means block until all data is done
+finished = process.isFinalOutputProcessed();
+ } else {
// there is some output
+finished = !process.isRecentOutputEmpty();
+ }
+ if (finished) {
break;
}
try {
@@ -299,5 +302,4 @@ public class ForkedProcessService extends
AbstractWorkflowExecutorService implem
return process.getRecentOutput();
}
-
}
git commit: Fix typo in app-packages/hbase/README.txt
Repository: incubator-slider
Updated Branches:
refs/heads/develop 9a54a0b8d -> f53db8725
Fix typo in app-packages/hbase/README.txt
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/f53db872
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/f53db872
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/f53db872
Branch: refs/heads/develop
Commit: f53db87257f8a600594446f1a60a9614ec65369f
Parents: 9a54a0b
Author: tedyu
Authored: Tue Jul 1 12:37:35 2014 -0700
Committer: tedyu
Committed: Tue Jul 1 12:37:35 2014 -0700
--
app-packages/hbase/README.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f53db872/app-packages/hbase/README.txt
--
diff --git a/app-packages/hbase/README.txt b/app-packages/hbase/README.txt
index 78d033e..a25ff8c 100644
--- a/app-packages/hbase/README.txt
+++ b/app-packages/hbase/README.txt
@@ -30,7 +30,7 @@ Use the following command to install HBase tarball locally:
After HBase tarball is publised locally in maven repository, you can use the
following command:
mvn clean package -DskipTests -Phbase-app-package
App package can be found in
-
app-packages/HBase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip
+
app-packages/hbase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip
Create a zip package at the root of the package (/app-packages/hbase/)
zip -r hbase-v098.zip .
[2/2] git commit: SLIDER-190. Add projects to create command logger package as well as Slider app package
SLIDER-190. Add projects to create command logger package as well as Slider app
package
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e879109a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e879109a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e879109a
Branch: refs/heads/develop
Commit: e879109a45f255d63e3d32d5f8bb6d40a26e38e9
Parents: f53db87
Author: Sumit Mohanty
Authored: Mon Jun 30 19:47:04 2014 -0700
Committer: Sumit Mohanty
Committed: Tue Jul 1 12:56:33 2014 -0700
--
app-packages/command-logger/README.txt | 22
.../command-logger/application-pkg/pom.xml | 83 ++
.../src/command_logger/README.txt | 18
.../src/command_logger/operations.log | 24 +
.../src/packages/tarball/all.xml| 35 ++
.../command-logger/slider-pkg/appConfig.json| 27 +
.../slider-pkg/configuration/cl-site.xml| 34 ++
.../command-logger/slider-pkg/metainfo.xml | 52 +
.../slider-pkg/package/scripts/cl.py| 89 +++
.../slider-pkg/package/scripts/params.py| 31 ++
.../package/templates/operations.log.j2 | 22
app-packages/command-logger/slider-pkg/pom.xml | 107 +++
.../command-logger/slider-pkg/resources.json| 15 +++
.../slider-pkg/src/assembly/command-logger.xml | 78 ++
app-packages/hbase/appConfig.json | 4 +-
pom.xml | 2 +
16 files changed, 641 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/README.txt
--
diff --git a/app-packages/command-logger/README.txt
b/app-packages/command-logger/README.txt
new file mode 100644
index 000..56a8a33
--- /dev/null
+++ b/app-packages/command-logger/README.txt
@@ -0,0 +1,22 @@
+
+Command logger is a test application used by Slider for functional tests.
+
+Like any typical Slider Application, there is a native application package
+in the form of a tarball that is created by project under application_pkg and
+there is a project under slider_pkg to create the Slider Application Package
+that is consumed by Slider.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/application-pkg/pom.xml
--
diff --git a/app-packages/command-logger/application-pkg/pom.xml
b/app-packages/command-logger/application-pkg/pom.xml
new file mode 100644
index 000..7da7259
--- /dev/null
+++ b/app-packages/command-logger/application-pkg/pom.xml
@@ -0,0 +1,83 @@
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+
+org.apache.slider
+slider
+0.31.0-incubating-SNAPSHOT
+../../../pom.xml
+
+ 4.0.0
+ command-logger
+ pom
+ Command Logger
+ Command Logger
+
+command-logger
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-assembly-plugin
+
+ gnu
+ src/packages/tarball/all.xml
+ false
+ ${app.package.name}
+
+
+
+build-tarball
+package
+
+ single
+
+
+
+
+
+
+maven-compiler-plugin
+3.0
+
+
+
+org.apache.rat
+apache-rat-plugin
+${apache-rat-plugin.version}
+
+
+check-licenses
+
+ check
+
+
+
+
+
+
+
+org.apache.maven.wagon
+wagon-ssh-external
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/application-pkg/src/command_logger/README.txt
--
diff --git
a/app-packages/command-logger/application-pkg/src/command_logger/README.txt
b/app-packages/command-logger/application-pkg/src/command_logger/README.txt
new file mode 100644
index 000..19d2e26
--- /dev/null
+++ b/app-packages/command-logger/application-pkg/src/command_logger/README.txt
@@ -0,0 +1,18 @@
+
+
+All this project contains is a tarball that has a sample log file.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/application-pkg/src/command_logger/operations.log
--
diff --git
a/ap
[1/2] git commit: SLIDER-190. Add projects to create command logger package as well as Slider app package
Repository: incubator-slider
Updated Branches:
refs/heads/develop f53db8725 -> be94df3b4
SLIDER-190. Add projects to create command logger package as well as Slider app
package
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/be94df3b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/be94df3b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/be94df3b
Branch: refs/heads/develop
Commit: be94df3b49051dd929dd6dbb2736f9a0daac73b4
Parents: e879109
Author: Sumit Mohanty
Authored: Tue Jul 1 11:25:30 2014 -0700
Committer: Sumit Mohanty
Committed: Tue Jul 1 12:56:33 2014 -0700
--
.../command-logger/application-pkg/README.txt | 24 +
.../command-logger/slider-pkg/metainfo.xml | 2 +-
app-packages/command-logger/slider-pkg/pom.xml | 17 +++-
.../test_command_log/appConfig.json | 6 +-
.../test_command_log/appConfig_fast_no_reg.json | 6 +-
.../test_command_log/appConfig_no_hb.json | 6 +-
.../test_command_log/configuration/cl-site.xml | 34 ---
.../app_packages/test_command_log/metainfo.xml | 52 ---
.../package/files/command_log_10.tar| Bin 2560 -> 0 bytes
.../test_command_log/package/scripts/cl.py | 89 ---
.../test_command_log/package/scripts/params.py | 31 ---
.../package/templates/operations.log.j2 | 22 -
slider-funtest/pom.xml | 52 ++-
.../lifecycle/AgentCommandTestBase.groovy | 21 ++---
14 files changed, 107 insertions(+), 255 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/app-packages/command-logger/application-pkg/README.txt
--
diff --git a/app-packages/command-logger/application-pkg/README.txt
b/app-packages/command-logger/application-pkg/README.txt
new file mode 100644
index 000..5596f30
--- /dev/null
+++ b/app-packages/command-logger/application-pkg/README.txt
@@ -0,0 +1,24 @@
+
+Command logger is a simple application that does not have any moving parts yet.
+Its sole purpose is to get packaged as a tarball that includes some static
files
+suggesting how commands may be logged.
+
+TODO:
+* Add a daemon that gets activated when application is STARTED
+* Provide hooks to validate that daemon is behaving as expected
+* Provide hooks to inject failures
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/app-packages/command-logger/slider-pkg/metainfo.xml
--
diff --git a/app-packages/command-logger/slider-pkg/metainfo.xml
b/app-packages/command-logger/slider-pkg/metainfo.xml
index 845ad93..e17413d 100644
--- a/app-packages/command-logger/slider-pkg/metainfo.xml
+++ b/app-packages/command-logger/slider-pkg/metainfo.xml
@@ -42,7 +42,7 @@
tarball
-files/command_log.tar.gz
+files/command_log.tar
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/app-packages/command-logger/slider-pkg/pom.xml
--
diff --git a/app-packages/command-logger/slider-pkg/pom.xml
b/app-packages/command-logger/slider-pkg/pom.xml
index 72af03e..42ec8da 100644
--- a/app-packages/command-logger/slider-pkg/pom.xml
+++ b/app-packages/command-logger/slider-pkg/pom.xml
@@ -24,7 +24,7 @@
../../../pom.xml
4.0.0
- command-logger-app-package
+ apache-slider-command-logger
pom
Slider Command Logger App Package
Slider Command Logger App Package
@@ -92,6 +92,21 @@
+
+
+org.apache.rat
+apache-rat-plugin
+${apache-rat-plugin.version}
+
+
+check-licenses
+
+ check
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/slider-core/src/test/app_packages/test_command_log/appConfig.json
--
diff --git a/slider-core/src/test/app_packages/test_command_log/appConfig.json
b/slider-core/src/test/app_packages/test_command_log/appConfig.json
index bd76c05..e7f9700 100644
--- a/slider-core/src/test/app_packages/test_command_log/appConfig.json
+++ b/slider-core/src/test/app_packages/test_command_log/appConfig.json
@@ -4,15 +4,15 @@
},
"global": {
"agent.conf": "agent.ini",
-"application.def": "cmd_log_app_pkg.zip",
+"application.def": "apache-slider-command-logger.zip",
"config_types": "cl-site",
"java_home": "/usr/jdk64/jdk1.7.0_45",
-"packag
[1/8] git commit: SLIDER-171 create config for windows VM "morzine"
Repository: incubator-slider
Updated Branches:
refs/heads/develop be94df3b4 -> e4723f5e5
SLIDER-171 create config for windows VM "morzine"
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0060d48b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0060d48b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0060d48b
Branch: refs/heads/develop
Commit: 0060d48bda1f18f181029bf49d93be7e25c69e96
Parents: 9a54a0b
Author: Steve Loughran
Authored: Tue Jul 1 18:49:16 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 18:49:16 2014 +0100
--
.../clusters/morzine/slider/log4j.properties| 83
.../clusters/morzine/slider/slider-client.xml | 59 ++
2 files changed, 142 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0060d48b/src/test/clusters/morzine/slider/log4j.properties
--
diff --git a/src/test/clusters/morzine/slider/log4j.properties
b/src/test/clusters/morzine/slider/log4j.properties
new file mode 100644
index 000..6211771
--- /dev/null
+++ b/src/test/clusters/morzine/slider/log4j.properties
@@ -0,0 +1,83 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License. See accompanying LICENSE file.
+#
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# log4j configuration used during build and unit tests
+
+log4j.rootLogger=INFO,stdout
+log4j.threshhold=ALL
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+
+# log layout skips stack-trace creation operations by avoiding line numbers
and method
+#log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} -
%m%n
+
+# debug edition is much more expensive
+log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2}
(%F:%M(%L)) - %m%n
+
+
+log4j.appender.subprocess=org.apache.log4j.ConsoleAppender
+log4j.appender.subprocess.layout=org.apache.log4j.PatternLayout
+log4j.appender.subprocess.layout.ConversionPattern=[%c{1}]: %m%n
+
+log4j.logger.org.apache.slider=DEBUG
+
+# uncomment to debug service lifecycle issues
+#log4j.logger.org.apache.hadoop.yarn.service.launcher=DEBUG
+#log4j.logger.org.apache.hadoop.yarn.service=DEBUG
+
+# uncomment for YARN operations
+#log4j.logger.org.apache.hadoop.yarn.client=DEBUG
+
+# uncomment this to debug security problems
+#log4j.logger.org.apache.hadoop.security=DEBUG
+
+#crank back on some noise
+log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
+log4j.logger.org.apache.hadoop.hdfs=WARN
+
+
+log4j.logger.org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor=WARN
+log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl=WARN
+log4j.logger.org.apache.zookeeper=WARN
+
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0060d48b/src/test/clusters/m
[4/8] git commit: SLIDER-171 more details on dir owner for diags
SLIDER-171 more details on dir owner for diags
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/abb3d019
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/abb3d019
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/abb3d019
Branch: refs/heads/develop
Commit: abb3d0190c55a79224e5b868844f72b82d28d71f
Parents: fb73dbc
Author: Steve Loughran
Authored: Tue Jul 1 20:15:51 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 20:15:51 2014 +0100
--
.../org/apache/slider/funtest/basic/TestClusterConnectivity.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/abb3d019/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
--
diff --git
a/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
index cf9cc9d..dcc7333 100644
---
a/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
+++
b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
@@ -36,7 +36,7 @@ class TestClusterConnectivity extends CommandTestBase {
def fs = clusterFS
def status = fs.listStatus(new Path("/"))
-status.each {it -> log.info(it.path.toString())}
+status.each {it -> log.info("${it.path} = ${it}")}
}
[8/8] git commit: Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop
Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e4723f5e Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e4723f5e Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e4723f5e Branch: refs/heads/develop Commit: e4723f5e538d15c39320c3651d6bf4c400c59400 Parents: 613dbec be94df3 Author: Steve Loughran Authored: Tue Jul 1 21:44:44 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 21:44:44 2014 +0100 -- app-packages/command-logger/README.txt | 22 .../command-logger/application-pkg/README.txt | 24 .../command-logger/application-pkg/pom.xml | 83 + .../src/command_logger/README.txt | 18 +++ .../src/command_logger/operations.log | 24 .../src/packages/tarball/all.xml| 35 ++ .../command-logger/slider-pkg/appConfig.json| 27 .../slider-pkg/configuration/cl-site.xml| 34 ++ .../command-logger/slider-pkg/metainfo.xml | 52 .../slider-pkg/package/scripts/cl.py| 89 ++ .../slider-pkg/package/scripts/params.py| 31 + .../package/templates/operations.log.j2 | 22 app-packages/command-logger/slider-pkg/pom.xml | 122 +++ .../command-logger/slider-pkg/resources.json| 15 +++ .../slider-pkg/src/assembly/command-logger.xml | 78 app-packages/hbase/README.txt | 2 +- app-packages/hbase/appConfig.json | 4 +- pom.xml | 2 + .../test_command_log/appConfig.json | 6 +- .../test_command_log/appConfig_fast_no_reg.json | 6 +- .../test_command_log/appConfig_no_hb.json | 6 +- .../test_command_log/configuration/cl-site.xml | 34 -- .../app_packages/test_command_log/metainfo.xml | 52 .../package/files/command_log_10.tar| Bin 2560 -> 0 bytes .../test_command_log/package/scripts/cl.py | 89 -- .../test_command_log/package/scripts/params.py | 31 - .../package/templates/operations.log.j2 | 22 slider-funtest/pom.xml | 52 +++- .../lifecycle/AgentCommandTestBase.groovy | 21 +--- 29 files changed, 747 insertions(+), 256 deletions(-) --
[6/8] git commit: Merge branch 'feature/SLIDER-171-slider-on-windows' into develop
Merge branch 'feature/SLIDER-171-slider-on-windows' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0317c411 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0317c411 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0317c411 Branch: refs/heads/develop Commit: 0317c411590e378ce8dc03e1b13b7eb1acd0ea32 Parents: 9a54a0b 853a979 Author: Steve Loughran Authored: Tue Jul 1 21:10:16 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 21:10:16 2014 +0100 -- .../slider/client/SliderYarnClientImpl.java | 3 +- .../apache/slider/common/tools/SliderUtils.java | 2 +- .../core/launch/ClasspathConstructor.java | 4 +- .../core/launch/JavaCommandLineBuilder.java | 2 +- .../funtest/framework/CommandTestBase.groovy| 4 +- .../funtest/framework/FileUploader.groovy | 30 -- .../basic/TestClusterConnectivity.groovy| 96 .../lifecycle/TestAgentClusterLifecycle.groovy | 3 +- .../clusters/c6401/slider/slider-client.xml | 19 ++-- .../clusters/morzine/slider/log4j.properties| 83 + .../clusters/morzine/slider/slider-client.xml | 76 .../clusters/offline/slider/slider-client.xml | 7 +- .../clusters/remote/slider/slider-client.xml| 5 - .../clusters/sandbox/slider/slider-client.xml | 5 - src/test/clusters/ubuntu-secure/operations.md | 36 15 files changed, 315 insertions(+), 60 deletions(-) --
[3/8] git commit: SLIDER-171: connectivity tests to verify state of funtest client config
SLIDER-171: connectivity tests to verify state of funtest client config
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/fb73dbca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/fb73dbca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/fb73dbca
Branch: refs/heads/develop
Commit: fb73dbca80cd87c2880d7e46325a67c507f58ee0
Parents: a6a4680
Author: Steve Loughran
Authored: Tue Jul 1 20:08:54 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 20:08:54 2014 +0100
--
.../slider/client/SliderYarnClientImpl.java | 3 +-
.../funtest/framework/CommandTestBase.groovy| 4 +-
.../basic/TestClusterConnectivity.groovy| 96
.../clusters/c6401/slider/slider-client.xml | 14 ++-
.../clusters/morzine/slider/slider-client.xml | 29 --
5 files changed, 132 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb73dbca/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java
b/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java
index 0c83e0c..e7b492b 100644
---
a/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java
+++
b/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java
@@ -19,6 +19,7 @@
package org.apache.slider.client;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationRequest;
@@ -74,7 +75,7 @@ public class SliderYarnClientImpl extends YarnClientImpl {
List allApps = getApplications(types);
List results = new ArrayList<>();
for (ApplicationReport report : allApps) {
- if (user == null || user.equals(report.getUser())) {
+ if (StringUtils.isEmpty(user) || user.equals(report.getUser())) {
results.add(report);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb73dbca/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
--
diff --git
a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
index 61ae804..08d352a 100644
---
a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
+++
b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
@@ -103,15 +103,13 @@ abstract class CommandTestBase extends SliderTestUtils {
log.debug("Security enabled")
SliderUtils.forceLogin()
} else {
- log.info "Security off, making cluster dirs broadly accessible"
+ log.info "Security is off"
}
SliderShell.confDir = SLIDER_CONF_DIRECTORY
SliderShell.script = SLIDER_SCRIPT
log.info("Test using ${HadoopFS.getDefaultUri(SLIDER_CONFIG)} " +
"and YARN RM @
${SLIDER_CONFIG.get(YarnConfiguration.RM_ADDRESS)}")
-// now patch the settings with the path of the conf direcotry
-
}
/**
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb73dbca/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
--
diff --git
a/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
new file mode 100644
index 000..cf9cc9d
--- /dev/null
+++
b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either e
[5/8] git commit: SLIDER-171 changes to setting up java command and classpath needed to work with cross-platform client
SLIDER-171 changes to setting up java command and classpath needed to work with
cross-platform client
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/853a9790
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/853a9790
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/853a9790
Branch: refs/heads/develop
Commit: 853a979006d5a08734a63ff80ce9c15ab577d61e
Parents: abb3d01
Author: Steve Loughran
Authored: Tue Jul 1 21:09:51 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 21:09:51 2014 +0100
--
.../apache/slider/common/tools/SliderUtils.java | 2 +-
.../core/launch/ClasspathConstructor.java | 4 +--
.../core/launch/JavaCommandLineBuilder.java | 2 +-
.../funtest/framework/FileUploader.groovy | 30 ++--
.../lifecycle/TestAgentClusterLifecycle.groovy | 3 +-
5 files changed, 28 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index 5479b54..62e61c9 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -1146,7 +1146,7 @@ public final class SliderUtils {
// and nothing else
classpath.appendAll(classpath.javaVMClasspath());
} else {
- classpath.addLibDir("./" + libdir);
+ classpath.addLibDir(libdir);
if (sliderConfDir != null) {
classpath.addClassDirectory(sliderConfDir);
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
index ee4c4d1..ca325d1 100644
---
a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
+++
b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
@@ -38,8 +38,8 @@ import java.util.List;
*/
public class ClasspathConstructor {
- // public static final String CLASS_PATH_SEPARATOR =
ApplicationConstants.CLASS_PATH_SEPARATOR;
- public static final String CLASS_PATH_SEPARATOR = File.pathSeparator;
+public static final String CLASS_PATH_SEPARATOR =
ApplicationConstants.CLASS_PATH_SEPARATOR;
+// public static final String CLASS_PATH_SEPARATOR = File.pathSeparator;
private final List pathElements = new ArrayList<>();
public ClasspathConstructor() {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
b/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
index e8b6eb1..7b60461 100644
---
a/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
+++
b/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
@@ -38,7 +38,7 @@ public class JavaCommandLineBuilder extends
CommandLineBuilder {
* @return the path to the Java binary
*/
protected String getJavaBinary() {
-return ApplicationConstants.Environment.JAVA_HOME.$() + "/bin/java";
+return ApplicationConstants.Environment.JAVA_HOME.$$() + "/bin/java";
}
/**
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy
--
diff --git
a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy
b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy
index 2dc85be..921adbf 100644
---
a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy
+++
b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy
@@ -18,6 +18,7 @@
package org.apache.slider.funtest.framework
+import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem as
[2/8] git commit: SLIDER-192 drop obsolete references to slider.security.enabled
SLIDER-192 drop obsolete references to slider.security.enabled Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a6a4680f Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a6a4680f Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a6a4680f Branch: refs/heads/develop Commit: a6a4680fec58330e5377c05b60061231e0be4037 Parents: 0060d48 Author: Steve Loughran Authored: Tue Jul 1 18:50:01 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 18:50:01 2014 +0100 -- .../clusters/c6401/slider/slider-client.xml | 5 --- .../clusters/offline/slider/slider-client.xml | 7 +--- .../clusters/remote/slider/slider-client.xml| 5 --- .../clusters/sandbox/slider/slider-client.xml | 5 --- src/test/clusters/ubuntu-secure/operations.md | 36 ++-- 5 files changed, 19 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/c6401/slider/slider-client.xml -- diff --git a/src/test/clusters/c6401/slider/slider-client.xml b/src/test/clusters/c6401/slider/slider-client.xml index f1aa4fb..fe2caad 100644 --- a/src/test/clusters/c6401/slider/slider-client.xml +++ b/src/test/clusters/c6401/slider/slider-client.xml @@ -38,11 +38,6 @@ -slider.security.enabled -false - - - slider.zookeeper.quorum c6401:2181 http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/offline/slider/slider-client.xml -- diff --git a/src/test/clusters/offline/slider/slider-client.xml b/src/test/clusters/offline/slider/slider-client.xml index a49dd29..8385086 100644 --- a/src/test/clusters/offline/slider/slider-client.xml +++ b/src/test/clusters/offline/slider/slider-client.xml @@ -41,12 +41,7 @@ slider.funtest.enabled false - - -slider.security.enabled -false - - + yarn.application.classpath http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/remote/slider/slider-client.xml -- diff --git a/src/test/clusters/remote/slider/slider-client.xml b/src/test/clusters/remote/slider/slider-client.xml index 5bd2edb..5ed4d10 100644 --- a/src/test/clusters/remote/slider/slider-client.xml +++ b/src/test/clusters/remote/slider/slider-client.xml @@ -50,11 +50,6 @@ -slider.security.enabled -false - - - slider.test.agent.enabled true http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/sandbox/slider/slider-client.xml -- diff --git a/src/test/clusters/sandbox/slider/slider-client.xml b/src/test/clusters/sandbox/slider/slider-client.xml index f15f71d..30937ec 100644 --- a/src/test/clusters/sandbox/slider/slider-client.xml +++ b/src/test/clusters/sandbox/slider/slider-client.xml @@ -38,11 +38,6 @@ -slider.security.enabled -false - - - slider.zookeeper.quorum sandbox:2181 http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/ubuntu-secure/operations.md -- diff --git a/src/test/clusters/ubuntu-secure/operations.md b/src/test/clusters/ubuntu-secure/operations.md index cb14dfe..d894038 100644 --- a/src/test/clusters/ubuntu-secure/operations.md +++ b/src/test/clusters/ubuntu-secure/operations.md @@ -56,7 +56,7 @@ --manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 \ --role workers 4\ --zkhosts ubuntu --zkport 2121 \ - -D slider.security.enabled=true -S java.security.krb5.realm=COTHAM \ + -S java.security.krb5.realm=COTHAM \ -S java.security.krb5.kdc=ubuntu \ --image hdfs://ubuntu:9090/hbase.tar \ --appconf file:Users/slider/Hadoop/configs/master/hbase \ @@ -70,7 +70,7 @@ bin/slider create cl1 \ --provider hbase \ --manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 \ --D slider.security.enabled=true -S java.security.krb5.realm=COTHAM \ +-S java.security.krb5.realm=COTHAM \ -S java.security.krb5.kdc=ubuntu \ -D yarn.resourcemanager.principal=yarn/ubuntu@COTHAM \ --role worker 1\ @@ -87,7 +87,7 @@ bin/slider create cl1 \ --provider hbase \ --manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 \ --D slider.security.enabled=true -S java.security.krb5.realm=COTHAM \ +-S java.security.krb5.realm=COTHAM \ -S java.security.krb5.kdc=ubu
[7/8] git commit: SLIDER-171: mark up some methods as clearly not for use x-platform, but only for local setup of miniclusters
SLIDER-171: mark up some methods as clearly not for use x-platform, but only
for local setup of miniclusters
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/613dbec8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/613dbec8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/613dbec8
Branch: refs/heads/develop
Commit: 613dbec829810edff005c9b8aee00eba8f8bc1b8
Parents: 0317c41
Author: Steve Loughran
Authored: Tue Jul 1 21:44:35 2014 +0100
Committer: Steve Loughran
Committed: Tue Jul 1 21:44:35 2014 +0100
--
.../org/apache/slider/common/tools/SliderUtils.java | 2 +-
.../slider/core/launch/ClasspathConstructor.java| 16 +---
2 files changed, 14 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/613dbec8/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index 62e61c9..7c89321 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -1144,7 +1144,7 @@ public final class SliderUtils {
if (usingMiniMRCluster) {
// for mini cluster we pass down the java CP properties
// and nothing else
- classpath.appendAll(classpath.javaVMClasspath());
+ classpath.appendAll(classpath.localJVMClasspath());
} else {
classpath.addLibDir(libdir);
if (sliderConfDir != null) {
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/613dbec8/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
index ca325d1..5b74c03 100644
---
a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
+++
b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java
@@ -123,12 +123,12 @@ public class ClasspathConstructor {
public void addRemoteClasspathEnvVar() {
-append(ApplicationConstants.Environment.CLASSPATH.$());
+append(ApplicationConstants.Environment.CLASSPATH.$$());
}
public void insertRemoteClasspathEnvVar() {
-append(ApplicationConstants.Environment.CLASSPATH.$());
+append(ApplicationConstants.Environment.CLASSPATH.$$());
}
@@ -152,12 +152,22 @@ public class ClasspathConstructor {
return dir;
}
+ /**
+ * Split a classpath. This uses the local path separator so MUST NOT
+ * be used to work with remote classpaths
+ * @param localpath local path
+ * @return a splite
+ */
public Collection splitClasspath(String localpath) {
String separator = System.getProperty("path.separator");
return StringUtils.getStringCollection(localpath, separator);
}
- public Collection javaVMClasspath() {
+ /**
+ * Get the local JVM classpath split up
+ * @return the list of entries on the JVM classpath env var
+ */
+ public Collection localJVMClasspath() {
return splitClasspath(System.getProperty("java.class.path"));
}
git commit: SLIDER-199. Modify HBase package README to also add a manual way to create .zip packages
Repository: incubator-slider
Updated Branches:
refs/heads/develop e4723f5e5 -> 5bb80a42b
SLIDER-199. Modify HBase package README to also add a manual way to create .zip
packages
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/5bb80a42
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/5bb80a42
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/5bb80a42
Branch: refs/heads/develop
Commit: 5bb80a42b1f5d6ea0c37d54b9dcb3f2ebf9251e1
Parents: e4723f5
Author: Sumit Mohanty
Authored: Tue Jul 1 17:42:43 2014 -0700
Committer: Sumit Mohanty
Committed: Tue Jul 1 17:42:48 2014 -0700
--
app-packages/hbase/README.txt | 54 +-
pom.xml | 1 -
2 files changed, 36 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5bb80a42/app-packages/hbase/README.txt
--
diff --git a/app-packages/hbase/README.txt b/app-packages/hbase/README.txt
index a25ff8c..1d547a2 100644
--- a/app-packages/hbase/README.txt
+++ b/app-packages/hbase/README.txt
@@ -15,40 +15,58 @@
limitations under the License.
-->
-How to create a Slider package for HBase?
+Create Slider App Package for HBase
+
+While appConfig.json and resources.json are not required for the package they
+work well as the default configuration for Slider apps. So it is advisable that
+when you create an application package for Slider, include sample/default
+resources.json and appConfig.json for a minimal Yarn cluster.
+
+OPTION-I: Use mvn command
+OPTION-II: Manual
+
+** OPTION - I (use mvn command) **
+You need the HBase version available on local maven repo to create the Slider
App Package for HBase.
The version of HBase used for the app package can be adjusted by adding a
flag such as
-Dhbase.version=0.98.3
-Copy the tarball for HBase:
- cp ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz package/files/
+Download the tarball for HBase:
+ e.g. path to tarball ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz
Use the following command to install HBase tarball locally:
mvn install:install-file -Dfile= -DgroupId=org.apache.hbase
-DartifactId=hbase -Dversion=0.98.3-hadoop2 -Dclassifier=bin -Dpackaging=tar.gz
-After HBase tarball is publised locally in maven repository, you can use the
following command:
+After HBase tarball is published locally in maven repository, you can use the
following command:
mvn clean package -DskipTests -Phbase-app-package
+
App package can be found in
app-packages/hbase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip
-Create a zip package at the root of the package (/app-packages/hbase/)
- zip -r hbase-v098.zip .
-
Verify the content using
zip -Tv apache-slider-hbase-*.zip
-While appConfig.json and resources.json are not required for the package they
-work well as the default configuration for Slider apps. So it is advisable that
-when you create an application package for Slider, include sample/default
-resources.json and appConfig.json for a minimal Yarn cluster.
+If an HBase version older than 0.98.3 is desired, it must be installed in the
local maven repo.
-If an HBase version older than 0.98.3 is desired, it must be installed in the
-local maven repo.
+A less descriptive file name can be specified with
+ -Dapp.package.name=HBase_98dot3 which would create a file HBase_98dot3.zip.
-**Note that the LICENSE.txt and NOTICE.txt that are bundled with the app
-package are designed for HBase 0.98.3 only and may need to be modified to be
-applicable for other versions of the app package.
+** OPTION - II (manual) **
+The Slider App Package for HBase can also be created manually.
-A less descriptive file name can be specified with
--Dapp.package.name=HBase_98dot3 which would create a file HBase_98dot3.zip.
+Download the tarball for HBase:
+ e.g. path to tarball ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz
+
+Copy the hbase tarball to package/files
+ cp ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz package/files
+
+Edit appConfig.json/metainfo.xml
+ Replace 4 occurrences of "${hbase.version}" with the hbase version values
such as "0.98.3-hadoop2"
+ Replace 1 occurrence of "${app.package.name}" with the desired app package
name, e.g. "hbase-v098"
+
+Create a zip package at the root of the package (/app-packages/hbase/)
+ zip -r hbase-v098.zip .
+
+Verify the content using
+ zip -Tv hbase-v098.zip
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5bb80a42/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e1f772b..460fb0d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -39,7 +39,6 @@
