svn commit: r1607095 - /incubator/slider/site/trunk/content/docs/security.md
Author: stevel Date: Tue Jul 1 14:25:30 2014 New Revision: 1607095 URL: http://svn.apache.org/r1607095 Log: SLIDER-192: use hadoop.security.authentication type as indicate whether to use security or not. Modified: incubator/slider/site/trunk/content/docs/security.md Modified: incubator/slider/site/trunk/content/docs/security.md URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/security.md?rev=1607095&r1=1607094&r2=1607095&view=diff == --- incubator/slider/site/trunk/content/docs/security.md (original) +++ incubator/slider/site/trunk/content/docs/security.md Tue Jul 1 14:25:30 2014 @@ -24,7 +24,7 @@ to deploy secure applications on a secur This document does not cover Kerberos, how to secure a Hadoop cluster, Kerberos command line tools or how Hadoop uses delegation tokens to delegate permissions -round a cluster. These are assumed, though some links to useful pages are +round a cluster. These are prerequisites &em;though some links to useful pages are listed at the bottom. @@ -72,20 +72,21 @@ Slider runs in secure clusters, but with * Slider application instance and HBase instance to remain functional and secure over an indefinite period of time. ### Initial Non-requirements -* secure audit trail of cluster operations. -* multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance. -* More than one HBase cluster in the YARN cluster belonging to a single user (irrespective of how they are started). +* Secure audit trail of cluster operations. +* Multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance. * Any way to revoke certificates/rights of running containers. ### Assumptions * Kerberos is running and that HDFS and YARN are running Kerberized. * LDAP cannot be assumed. -* Credentials needed for HBase can be pushed out into the local filesystems of +* Credentials needed for the application can be pushed out into the local filesystems of the of the worker nodes via some external mechanism (e.g. scp), and protected by the access permissions of the native filesystem. Any user with access to these credentials is considered to have been granted such rights. -* These credentials can outlive the duration of the HBase containers -* The user running HBase has the same identity as that of the HBase cluster. +* These credentials can outlive the duration of the application instances +* The user running the application has the same identity as that of the application. +* All application instances run by a single user can share the same machine-specific +kerberos identities. ## Design @@ -172,7 +173,7 @@ They can also be set on the Slider comma -S java.security.krb5.realm=MINICLUSTER -S java.security.krb5.kdc=hadoop-kdc -### Java Cryptography Exceptions +### Important: Java Cryptography Package When trying to talk to a secure, cluster you may see the message: @@ -184,6 +185,8 @@ needed to work with the keys that Kerber from Oracle (or other supplier of the JVM) and installed according to its accompanying instructions. + + ## Useful Links 1. [Adding Security to Apache Hadoop](http://hortonworks.com/wp-content/uploads/2011/10/security-design_withCover-1.pdf)
git commit: SLIDER-192: use hadoop.security.authentication type as indicate whether to use security or not.
Repository: incubator-slider Updated Branches: refs/heads/develop 31d22b888 -> 8fe71f067 SLIDER-192: use hadoop.security.authentication type as indicate whether to use security or not. Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8fe71f06 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8fe71f06 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8fe71f06 Branch: refs/heads/develop Commit: 8fe71f06738321525501b7c13a50046bf85e43fb Parents: 31d22b8 Author: Steve Loughran Authored: Tue Jul 1 15:24:58 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 15:24:58 2014 +0100 -- .../java/org/apache/slider/client/SliderClient.java | 4 +++- .../org/apache/slider/common/SliderXmlConfKeys.java | 4 ++-- .../org/apache/slider/common/tools/SliderUtils.java | 15 --- 3 files changed, 13 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8fe71f06/slider-core/src/main/java/org/apache/slider/client/SliderClient.java -- diff --git a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java index 95c120c..a337533 100644 --- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java +++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java @@ -894,7 +894,9 @@ public class SliderClient extends AbstractSliderLaunchedService implements RunSe if (clusterSecure) { // if the cluster is secure, make sure that // the relevant security settings go over - addConfOptionToCLI(commandLine, config, KEY_SECURITY_ENABLED); +/* + addConfOptionToCLI(commandLine, config, KEY_SECURITY); +*/ addConfOptionToCLI(commandLine, config, DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY); http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8fe71f06/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java -- diff --git a/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java b/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java index 272ae6a..c7b8ea5 100644 --- a/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java +++ b/slider-core/src/main/java/org/apache/slider/common/SliderXmlConfKeys.java @@ -93,8 +93,8 @@ public interface SliderXmlConfKeys { * Flag which is set to indicate that security should be enabled * when talking to this cluster. */ - String KEY_SECURITY_ENABLED = - CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION; + String KEY_SECURITY = + CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION; /** * queue name http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8fe71f06/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java -- diff --git a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java index b97ff63..5479b54 100644 --- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java +++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java @@ -904,7 +904,8 @@ public final class SliderUtils { * @return true if the slider client/service should be in secure mode */ public static boolean isHadoopClusterSecure(Configuration conf) { -return conf.getBoolean(SliderXmlConfKeys.KEY_SECURITY_ENABLED, false); +return SecurityUtil.getAuthenticationMethod(conf) != + UserGroupInformation.AuthenticationMethod.SIMPLE; } /** @@ -953,22 +954,22 @@ public final class SliderUtils { conf.get(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION)); log.debug("hadoop.security.authorization={}", conf.get(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION)); -SecurityUtil.setAuthenticationMethod( -UserGroupInformation.AuthenticationMethod.KERBEROS, conf); +/*SecurityUtil.setAuthenticationMethod( +UserGroupInformation.AuthenticationMethod.KERBEROS, conf);*/ UserGroupInformation.setConfiguration(conf); UserGroupInformation authUser = UserGroupInformation.getCurrentUser(); log.debug("Authenticating as " + authUser.toString()); log.debug("Login user is {}", UserGroupInformation.getLoginUser()); if (!UserGroupInformation.isSecurityEnabled()) { throw new BadConfigException("Although secure mode is enabled," + -
git commit: SLIDER-159: jenkins testEcho intermittently failing
Repository: incubator-slider Updated Branches: refs/heads/develop 8fe71f067 -> 9a54a0b8d SLIDER-159: jenkins testEcho intermittently failing Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/9a54a0b8 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/9a54a0b8 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/9a54a0b8 Branch: refs/heads/develop Commit: 9a54a0b8d3e6d5f5f706407dfe4ba432419b90d2 Parents: 8fe71f0 Author: Steve Loughran Authored: Tue Jul 1 16:19:58 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 16:19:58 2014 +0100 -- .../services/workflow/ForkedProcessService.java | 14 -- 1 file changed, 8 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9a54a0b8/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java -- diff --git a/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java b/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java index 7e73005..ee68aa4 100644 --- a/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java +++ b/slider-core/src/main/java/org/apache/slider/server/services/workflow/ForkedProcessService.java @@ -281,12 +281,15 @@ public class ForkedProcessService extends AbstractWorkflowExecutorService implem } long start = System.currentTimeMillis(); while (System.currentTimeMillis() - start <= duration) { - if (finalOutput && process.isFinalOutputProcessed()) { -//end of stream, break -break; - } - if (!process.isRecentOutputEmpty()) { + boolean finished; + if (finalOutput) { +// final flag means block until all data is done +finished = process.isFinalOutputProcessed(); + } else { // there is some output +finished = !process.isRecentOutputEmpty(); + } + if (finished) { break; } try { @@ -299,5 +302,4 @@ public class ForkedProcessService extends AbstractWorkflowExecutorService implem return process.getRecentOutput(); } - }
git commit: Fix typo in app-packages/hbase/README.txt
Repository: incubator-slider Updated Branches: refs/heads/develop 9a54a0b8d -> f53db8725 Fix typo in app-packages/hbase/README.txt Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/f53db872 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/f53db872 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/f53db872 Branch: refs/heads/develop Commit: f53db87257f8a600594446f1a60a9614ec65369f Parents: 9a54a0b Author: tedyu Authored: Tue Jul 1 12:37:35 2014 -0700 Committer: tedyu Committed: Tue Jul 1 12:37:35 2014 -0700 -- app-packages/hbase/README.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f53db872/app-packages/hbase/README.txt -- diff --git a/app-packages/hbase/README.txt b/app-packages/hbase/README.txt index 78d033e..a25ff8c 100644 --- a/app-packages/hbase/README.txt +++ b/app-packages/hbase/README.txt @@ -30,7 +30,7 @@ Use the following command to install HBase tarball locally: After HBase tarball is publised locally in maven repository, you can use the following command: mvn clean package -DskipTests -Phbase-app-package App package can be found in - app-packages/HBase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip + app-packages/hbase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip Create a zip package at the root of the package (/app-packages/hbase/) zip -r hbase-v098.zip .
[2/2] git commit: SLIDER-190. Add projects to create command logger package as well as Slider app package
SLIDER-190. Add projects to create command logger package as well as Slider app package Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e879109a Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e879109a Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e879109a Branch: refs/heads/develop Commit: e879109a45f255d63e3d32d5f8bb6d40a26e38e9 Parents: f53db87 Author: Sumit Mohanty Authored: Mon Jun 30 19:47:04 2014 -0700 Committer: Sumit Mohanty Committed: Tue Jul 1 12:56:33 2014 -0700 -- app-packages/command-logger/README.txt | 22 .../command-logger/application-pkg/pom.xml | 83 ++ .../src/command_logger/README.txt | 18 .../src/command_logger/operations.log | 24 + .../src/packages/tarball/all.xml| 35 ++ .../command-logger/slider-pkg/appConfig.json| 27 + .../slider-pkg/configuration/cl-site.xml| 34 ++ .../command-logger/slider-pkg/metainfo.xml | 52 + .../slider-pkg/package/scripts/cl.py| 89 +++ .../slider-pkg/package/scripts/params.py| 31 ++ .../package/templates/operations.log.j2 | 22 app-packages/command-logger/slider-pkg/pom.xml | 107 +++ .../command-logger/slider-pkg/resources.json| 15 +++ .../slider-pkg/src/assembly/command-logger.xml | 78 ++ app-packages/hbase/appConfig.json | 4 +- pom.xml | 2 + 16 files changed, 641 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/README.txt -- diff --git a/app-packages/command-logger/README.txt b/app-packages/command-logger/README.txt new file mode 100644 index 000..56a8a33 --- /dev/null +++ b/app-packages/command-logger/README.txt @@ -0,0 +1,22 @@ + +Command logger is a test application used by Slider for functional tests. + +Like any typical Slider Application, there is a native application package +in the form of a tarball that is created by project under application_pkg and +there is a project under slider_pkg to create the Slider Application Package +that is consumed by Slider. http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/application-pkg/pom.xml -- diff --git a/app-packages/command-logger/application-pkg/pom.xml b/app-packages/command-logger/application-pkg/pom.xml new file mode 100644 index 000..7da7259 --- /dev/null +++ b/app-packages/command-logger/application-pkg/pom.xml @@ -0,0 +1,83 @@ + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + + +org.apache.slider +slider +0.31.0-incubating-SNAPSHOT +../../../pom.xml + + 4.0.0 + command-logger + pom + Command Logger + Command Logger + +command-logger + + + + + + +org.apache.maven.plugins +maven-assembly-plugin + + gnu + src/packages/tarball/all.xml + false + ${app.package.name} + + + +build-tarball +package + + single + + + + + + +maven-compiler-plugin +3.0 + + + +org.apache.rat +apache-rat-plugin +${apache-rat-plugin.version} + + +check-licenses + + check + + + + + + + +org.apache.maven.wagon +wagon-ssh-external + + + + http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/application-pkg/src/command_logger/README.txt -- diff --git a/app-packages/command-logger/application-pkg/src/command_logger/README.txt b/app-packages/command-logger/application-pkg/src/command_logger/README.txt new file mode 100644 index 000..19d2e26 --- /dev/null +++ b/app-packages/command-logger/application-pkg/src/command_logger/README.txt @@ -0,0 +1,18 @@ + + +All this project contains is a tarball that has a sample log file. http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e879109a/app-packages/command-logger/application-pkg/src/command_logger/operations.log -- diff --git a/ap
[1/2] git commit: SLIDER-190. Add projects to create command logger package as well as Slider app package
Repository: incubator-slider Updated Branches: refs/heads/develop f53db8725 -> be94df3b4 SLIDER-190. Add projects to create command logger package as well as Slider app package Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/be94df3b Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/be94df3b Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/be94df3b Branch: refs/heads/develop Commit: be94df3b49051dd929dd6dbb2736f9a0daac73b4 Parents: e879109 Author: Sumit Mohanty Authored: Tue Jul 1 11:25:30 2014 -0700 Committer: Sumit Mohanty Committed: Tue Jul 1 12:56:33 2014 -0700 -- .../command-logger/application-pkg/README.txt | 24 + .../command-logger/slider-pkg/metainfo.xml | 2 +- app-packages/command-logger/slider-pkg/pom.xml | 17 +++- .../test_command_log/appConfig.json | 6 +- .../test_command_log/appConfig_fast_no_reg.json | 6 +- .../test_command_log/appConfig_no_hb.json | 6 +- .../test_command_log/configuration/cl-site.xml | 34 --- .../app_packages/test_command_log/metainfo.xml | 52 --- .../package/files/command_log_10.tar| Bin 2560 -> 0 bytes .../test_command_log/package/scripts/cl.py | 89 --- .../test_command_log/package/scripts/params.py | 31 --- .../package/templates/operations.log.j2 | 22 - slider-funtest/pom.xml | 52 ++- .../lifecycle/AgentCommandTestBase.groovy | 21 ++--- 14 files changed, 107 insertions(+), 255 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/app-packages/command-logger/application-pkg/README.txt -- diff --git a/app-packages/command-logger/application-pkg/README.txt b/app-packages/command-logger/application-pkg/README.txt new file mode 100644 index 000..5596f30 --- /dev/null +++ b/app-packages/command-logger/application-pkg/README.txt @@ -0,0 +1,24 @@ + +Command logger is a simple application that does not have any moving parts yet. +Its sole purpose is to get packaged as a tarball that includes some static files +suggesting how commands may be logged. + +TODO: +* Add a daemon that gets activated when application is STARTED +* Provide hooks to validate that daemon is behaving as expected +* Provide hooks to inject failures http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/app-packages/command-logger/slider-pkg/metainfo.xml -- diff --git a/app-packages/command-logger/slider-pkg/metainfo.xml b/app-packages/command-logger/slider-pkg/metainfo.xml index 845ad93..e17413d 100644 --- a/app-packages/command-logger/slider-pkg/metainfo.xml +++ b/app-packages/command-logger/slider-pkg/metainfo.xml @@ -42,7 +42,7 @@ tarball -files/command_log.tar.gz +files/command_log.tar http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/app-packages/command-logger/slider-pkg/pom.xml -- diff --git a/app-packages/command-logger/slider-pkg/pom.xml b/app-packages/command-logger/slider-pkg/pom.xml index 72af03e..42ec8da 100644 --- a/app-packages/command-logger/slider-pkg/pom.xml +++ b/app-packages/command-logger/slider-pkg/pom.xml @@ -24,7 +24,7 @@ ../../../pom.xml 4.0.0 - command-logger-app-package + apache-slider-command-logger pom Slider Command Logger App Package Slider Command Logger App Package @@ -92,6 +92,21 @@ + + +org.apache.rat +apache-rat-plugin +${apache-rat-plugin.version} + + +check-licenses + + check + + + + + http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/be94df3b/slider-core/src/test/app_packages/test_command_log/appConfig.json -- diff --git a/slider-core/src/test/app_packages/test_command_log/appConfig.json b/slider-core/src/test/app_packages/test_command_log/appConfig.json index bd76c05..e7f9700 100644 --- a/slider-core/src/test/app_packages/test_command_log/appConfig.json +++ b/slider-core/src/test/app_packages/test_command_log/appConfig.json @@ -4,15 +4,15 @@ }, "global": { "agent.conf": "agent.ini", -"application.def": "cmd_log_app_pkg.zip", +"application.def": "apache-slider-command-logger.zip", "config_types": "cl-site", "java_home": "/usr/jdk64/jdk1.7.0_45", -"packag
[1/8] git commit: SLIDER-171 create config for windows VM "morzine"
Repository: incubator-slider Updated Branches: refs/heads/develop be94df3b4 -> e4723f5e5 SLIDER-171 create config for windows VM "morzine" Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0060d48b Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0060d48b Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0060d48b Branch: refs/heads/develop Commit: 0060d48bda1f18f181029bf49d93be7e25c69e96 Parents: 9a54a0b Author: Steve Loughran Authored: Tue Jul 1 18:49:16 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 18:49:16 2014 +0100 -- .../clusters/morzine/slider/log4j.properties| 83 .../clusters/morzine/slider/slider-client.xml | 59 ++ 2 files changed, 142 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0060d48b/src/test/clusters/morzine/slider/log4j.properties -- diff --git a/src/test/clusters/morzine/slider/log4j.properties b/src/test/clusters/morzine/slider/log4j.properties new file mode 100644 index 000..6211771 --- /dev/null +++ b/src/test/clusters/morzine/slider/log4j.properties @@ -0,0 +1,83 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. See accompanying LICENSE file. +# + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# log4j configuration used during build and unit tests + +log4j.rootLogger=INFO,stdout +log4j.threshhold=ALL +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout + +# log layout skips stack-trace creation operations by avoiding line numbers and method +#log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} - %m%n + +# debug edition is much more expensive +log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} (%F:%M(%L)) - %m%n + + +log4j.appender.subprocess=org.apache.log4j.ConsoleAppender +log4j.appender.subprocess.layout=org.apache.log4j.PatternLayout +log4j.appender.subprocess.layout.ConversionPattern=[%c{1}]: %m%n + +log4j.logger.org.apache.slider=DEBUG + +# uncomment to debug service lifecycle issues +#log4j.logger.org.apache.hadoop.yarn.service.launcher=DEBUG +#log4j.logger.org.apache.hadoop.yarn.service=DEBUG + +# uncomment for YARN operations +#log4j.logger.org.apache.hadoop.yarn.client=DEBUG + +# uncomment this to debug security problems +#log4j.logger.org.apache.hadoop.security=DEBUG + +#crank back on some noise +log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR +log4j.logger.org.apache.hadoop.hdfs=WARN + + +log4j.logger.org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor=WARN +log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl=WARN +log4j.logger.org.apache.zookeeper=WARN + + http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0060d48b/src/test/clusters/m
[4/8] git commit: SLIDER-171 more details on dir owner for diags
SLIDER-171 more details on dir owner for diags Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/abb3d019 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/abb3d019 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/abb3d019 Branch: refs/heads/develop Commit: abb3d0190c55a79224e5b868844f72b82d28d71f Parents: fb73dbc Author: Steve Loughran Authored: Tue Jul 1 20:15:51 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 20:15:51 2014 +0100 -- .../org/apache/slider/funtest/basic/TestClusterConnectivity.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/abb3d019/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy -- diff --git a/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy index cf9cc9d..dcc7333 100644 --- a/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy +++ b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy @@ -36,7 +36,7 @@ class TestClusterConnectivity extends CommandTestBase { def fs = clusterFS def status = fs.listStatus(new Path("/")) -status.each {it -> log.info(it.path.toString())} +status.each {it -> log.info("${it.path} = ${it}")} }
[8/8] git commit: Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop
Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e4723f5e Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e4723f5e Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e4723f5e Branch: refs/heads/develop Commit: e4723f5e538d15c39320c3651d6bf4c400c59400 Parents: 613dbec be94df3 Author: Steve Loughran Authored: Tue Jul 1 21:44:44 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 21:44:44 2014 +0100 -- app-packages/command-logger/README.txt | 22 .../command-logger/application-pkg/README.txt | 24 .../command-logger/application-pkg/pom.xml | 83 + .../src/command_logger/README.txt | 18 +++ .../src/command_logger/operations.log | 24 .../src/packages/tarball/all.xml| 35 ++ .../command-logger/slider-pkg/appConfig.json| 27 .../slider-pkg/configuration/cl-site.xml| 34 ++ .../command-logger/slider-pkg/metainfo.xml | 52 .../slider-pkg/package/scripts/cl.py| 89 ++ .../slider-pkg/package/scripts/params.py| 31 + .../package/templates/operations.log.j2 | 22 app-packages/command-logger/slider-pkg/pom.xml | 122 +++ .../command-logger/slider-pkg/resources.json| 15 +++ .../slider-pkg/src/assembly/command-logger.xml | 78 app-packages/hbase/README.txt | 2 +- app-packages/hbase/appConfig.json | 4 +- pom.xml | 2 + .../test_command_log/appConfig.json | 6 +- .../test_command_log/appConfig_fast_no_reg.json | 6 +- .../test_command_log/appConfig_no_hb.json | 6 +- .../test_command_log/configuration/cl-site.xml | 34 -- .../app_packages/test_command_log/metainfo.xml | 52 .../package/files/command_log_10.tar| Bin 2560 -> 0 bytes .../test_command_log/package/scripts/cl.py | 89 -- .../test_command_log/package/scripts/params.py | 31 - .../package/templates/operations.log.j2 | 22 slider-funtest/pom.xml | 52 +++- .../lifecycle/AgentCommandTestBase.groovy | 21 +--- 29 files changed, 747 insertions(+), 256 deletions(-) --
[6/8] git commit: Merge branch 'feature/SLIDER-171-slider-on-windows' into develop
Merge branch 'feature/SLIDER-171-slider-on-windows' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0317c411 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0317c411 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0317c411 Branch: refs/heads/develop Commit: 0317c411590e378ce8dc03e1b13b7eb1acd0ea32 Parents: 9a54a0b 853a979 Author: Steve Loughran Authored: Tue Jul 1 21:10:16 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 21:10:16 2014 +0100 -- .../slider/client/SliderYarnClientImpl.java | 3 +- .../apache/slider/common/tools/SliderUtils.java | 2 +- .../core/launch/ClasspathConstructor.java | 4 +- .../core/launch/JavaCommandLineBuilder.java | 2 +- .../funtest/framework/CommandTestBase.groovy| 4 +- .../funtest/framework/FileUploader.groovy | 30 -- .../basic/TestClusterConnectivity.groovy| 96 .../lifecycle/TestAgentClusterLifecycle.groovy | 3 +- .../clusters/c6401/slider/slider-client.xml | 19 ++-- .../clusters/morzine/slider/log4j.properties| 83 + .../clusters/morzine/slider/slider-client.xml | 76 .../clusters/offline/slider/slider-client.xml | 7 +- .../clusters/remote/slider/slider-client.xml| 5 - .../clusters/sandbox/slider/slider-client.xml | 5 - src/test/clusters/ubuntu-secure/operations.md | 36 15 files changed, 315 insertions(+), 60 deletions(-) --
[3/8] git commit: SLIDER-171: connectivity tests to verify state of funtest client config
SLIDER-171: connectivity tests to verify state of funtest client config Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/fb73dbca Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/fb73dbca Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/fb73dbca Branch: refs/heads/develop Commit: fb73dbca80cd87c2880d7e46325a67c507f58ee0 Parents: a6a4680 Author: Steve Loughran Authored: Tue Jul 1 20:08:54 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 20:08:54 2014 +0100 -- .../slider/client/SliderYarnClientImpl.java | 3 +- .../funtest/framework/CommandTestBase.groovy| 4 +- .../basic/TestClusterConnectivity.groovy| 96 .../clusters/c6401/slider/slider-client.xml | 14 ++- .../clusters/morzine/slider/slider-client.xml | 29 -- 5 files changed, 132 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb73dbca/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java -- diff --git a/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java b/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java index 0c83e0c..e7b492b 100644 --- a/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java +++ b/slider-core/src/main/java/org/apache/slider/client/SliderYarnClientImpl.java @@ -19,6 +19,7 @@ package org.apache.slider.client; import com.google.common.annotations.VisibleForTesting; +import org.apache.commons.lang.StringUtils; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.yarn.api.ApplicationClientProtocol; import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationRequest; @@ -74,7 +75,7 @@ public class SliderYarnClientImpl extends YarnClientImpl { List allApps = getApplications(types); List results = new ArrayList<>(); for (ApplicationReport report : allApps) { - if (user == null || user.equals(report.getUser())) { + if (StringUtils.isEmpty(user) || user.equals(report.getUser())) { results.add(report); } } http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb73dbca/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy -- diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy index 61ae804..08d352a 100644 --- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy +++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy @@ -103,15 +103,13 @@ abstract class CommandTestBase extends SliderTestUtils { log.debug("Security enabled") SliderUtils.forceLogin() } else { - log.info "Security off, making cluster dirs broadly accessible" + log.info "Security is off" } SliderShell.confDir = SLIDER_CONF_DIRECTORY SliderShell.script = SLIDER_SCRIPT log.info("Test using ${HadoopFS.getDefaultUri(SLIDER_CONFIG)} " + "and YARN RM @ ${SLIDER_CONFIG.get(YarnConfiguration.RM_ADDRESS)}") -// now patch the settings with the path of the conf direcotry - } /** http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb73dbca/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy -- diff --git a/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy new file mode 100644 index 000..cf9cc9d --- /dev/null +++ b/slider-funtest/src/test/groovy/org/apache/slider/funtest/basic/TestClusterConnectivity.groovy @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either e
[5/8] git commit: SLIDER-171 changes to setting up java command and classpath needed to work with cross-platform client
SLIDER-171 changes to setting up java command and classpath needed to work with cross-platform client Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/853a9790 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/853a9790 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/853a9790 Branch: refs/heads/develop Commit: 853a979006d5a08734a63ff80ce9c15ab577d61e Parents: abb3d01 Author: Steve Loughran Authored: Tue Jul 1 21:09:51 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 21:09:51 2014 +0100 -- .../apache/slider/common/tools/SliderUtils.java | 2 +- .../core/launch/ClasspathConstructor.java | 4 +-- .../core/launch/JavaCommandLineBuilder.java | 2 +- .../funtest/framework/FileUploader.groovy | 30 ++-- .../lifecycle/TestAgentClusterLifecycle.groovy | 3 +- 5 files changed, 28 insertions(+), 13 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java -- diff --git a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java index 5479b54..62e61c9 100644 --- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java +++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java @@ -1146,7 +1146,7 @@ public final class SliderUtils { // and nothing else classpath.appendAll(classpath.javaVMClasspath()); } else { - classpath.addLibDir("./" + libdir); + classpath.addLibDir(libdir); if (sliderConfDir != null) { classpath.addClassDirectory(sliderConfDir); } http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java -- diff --git a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java index ee4c4d1..ca325d1 100644 --- a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java +++ b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java @@ -38,8 +38,8 @@ import java.util.List; */ public class ClasspathConstructor { - // public static final String CLASS_PATH_SEPARATOR = ApplicationConstants.CLASS_PATH_SEPARATOR; - public static final String CLASS_PATH_SEPARATOR = File.pathSeparator; +public static final String CLASS_PATH_SEPARATOR = ApplicationConstants.CLASS_PATH_SEPARATOR; +// public static final String CLASS_PATH_SEPARATOR = File.pathSeparator; private final List pathElements = new ArrayList<>(); public ClasspathConstructor() { http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java -- diff --git a/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java b/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java index e8b6eb1..7b60461 100644 --- a/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java +++ b/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java @@ -38,7 +38,7 @@ public class JavaCommandLineBuilder extends CommandLineBuilder { * @return the path to the Java binary */ protected String getJavaBinary() { -return ApplicationConstants.Environment.JAVA_HOME.$() + "/bin/java"; +return ApplicationConstants.Environment.JAVA_HOME.$$() + "/bin/java"; } /** http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/853a9790/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy -- diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy index 2dc85be..921adbf 100644 --- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy +++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FileUploader.groovy @@ -18,6 +18,7 @@ package org.apache.slider.funtest.framework +import groovy.transform.CompileStatic import groovy.util.logging.Slf4j import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.FileSystem as
[2/8] git commit: SLIDER-192 drop obsolete references to slider.security.enabled
SLIDER-192 drop obsolete references to slider.security.enabled Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a6a4680f Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a6a4680f Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a6a4680f Branch: refs/heads/develop Commit: a6a4680fec58330e5377c05b60061231e0be4037 Parents: 0060d48 Author: Steve Loughran Authored: Tue Jul 1 18:50:01 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 18:50:01 2014 +0100 -- .../clusters/c6401/slider/slider-client.xml | 5 --- .../clusters/offline/slider/slider-client.xml | 7 +--- .../clusters/remote/slider/slider-client.xml| 5 --- .../clusters/sandbox/slider/slider-client.xml | 5 --- src/test/clusters/ubuntu-secure/operations.md | 36 ++-- 5 files changed, 19 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/c6401/slider/slider-client.xml -- diff --git a/src/test/clusters/c6401/slider/slider-client.xml b/src/test/clusters/c6401/slider/slider-client.xml index f1aa4fb..fe2caad 100644 --- a/src/test/clusters/c6401/slider/slider-client.xml +++ b/src/test/clusters/c6401/slider/slider-client.xml @@ -38,11 +38,6 @@ -slider.security.enabled -false - - - slider.zookeeper.quorum c6401:2181 http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/offline/slider/slider-client.xml -- diff --git a/src/test/clusters/offline/slider/slider-client.xml b/src/test/clusters/offline/slider/slider-client.xml index a49dd29..8385086 100644 --- a/src/test/clusters/offline/slider/slider-client.xml +++ b/src/test/clusters/offline/slider/slider-client.xml @@ -41,12 +41,7 @@ slider.funtest.enabled false - - -slider.security.enabled -false - - + yarn.application.classpath http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/remote/slider/slider-client.xml -- diff --git a/src/test/clusters/remote/slider/slider-client.xml b/src/test/clusters/remote/slider/slider-client.xml index 5bd2edb..5ed4d10 100644 --- a/src/test/clusters/remote/slider/slider-client.xml +++ b/src/test/clusters/remote/slider/slider-client.xml @@ -50,11 +50,6 @@ -slider.security.enabled -false - - - slider.test.agent.enabled true http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/sandbox/slider/slider-client.xml -- diff --git a/src/test/clusters/sandbox/slider/slider-client.xml b/src/test/clusters/sandbox/slider/slider-client.xml index f15f71d..30937ec 100644 --- a/src/test/clusters/sandbox/slider/slider-client.xml +++ b/src/test/clusters/sandbox/slider/slider-client.xml @@ -38,11 +38,6 @@ -slider.security.enabled -false - - - slider.zookeeper.quorum sandbox:2181 http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a6a4680f/src/test/clusters/ubuntu-secure/operations.md -- diff --git a/src/test/clusters/ubuntu-secure/operations.md b/src/test/clusters/ubuntu-secure/operations.md index cb14dfe..d894038 100644 --- a/src/test/clusters/ubuntu-secure/operations.md +++ b/src/test/clusters/ubuntu-secure/operations.md @@ -56,7 +56,7 @@ --manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 \ --role workers 4\ --zkhosts ubuntu --zkport 2121 \ - -D slider.security.enabled=true -S java.security.krb5.realm=COTHAM \ + -S java.security.krb5.realm=COTHAM \ -S java.security.krb5.kdc=ubuntu \ --image hdfs://ubuntu:9090/hbase.tar \ --appconf file:Users/slider/Hadoop/configs/master/hbase \ @@ -70,7 +70,7 @@ bin/slider create cl1 \ --provider hbase \ --manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 \ --D slider.security.enabled=true -S java.security.krb5.realm=COTHAM \ +-S java.security.krb5.realm=COTHAM \ -S java.security.krb5.kdc=ubuntu \ -D yarn.resourcemanager.principal=yarn/ubuntu@COTHAM \ --role worker 1\ @@ -87,7 +87,7 @@ bin/slider create cl1 \ --provider hbase \ --manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 \ --D slider.security.enabled=true -S java.security.krb5.realm=COTHAM \ +-S java.security.krb5.realm=COTHAM \ -S java.security.krb5.kdc=ubu
[7/8] git commit: SLIDER-171: mark up some methods as clearly not for use x-platform, but only for local setup of miniclusters
SLIDER-171: mark up some methods as clearly not for use x-platform, but only for local setup of miniclusters Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/613dbec8 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/613dbec8 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/613dbec8 Branch: refs/heads/develop Commit: 613dbec829810edff005c9b8aee00eba8f8bc1b8 Parents: 0317c41 Author: Steve Loughran Authored: Tue Jul 1 21:44:35 2014 +0100 Committer: Steve Loughran Committed: Tue Jul 1 21:44:35 2014 +0100 -- .../org/apache/slider/common/tools/SliderUtils.java | 2 +- .../slider/core/launch/ClasspathConstructor.java| 16 +--- 2 files changed, 14 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/613dbec8/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java -- diff --git a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java index 62e61c9..7c89321 100644 --- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java +++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java @@ -1144,7 +1144,7 @@ public final class SliderUtils { if (usingMiniMRCluster) { // for mini cluster we pass down the java CP properties // and nothing else - classpath.appendAll(classpath.javaVMClasspath()); + classpath.appendAll(classpath.localJVMClasspath()); } else { classpath.addLibDir(libdir); if (sliderConfDir != null) { http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/613dbec8/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java -- diff --git a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java index ca325d1..5b74c03 100644 --- a/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java +++ b/slider-core/src/main/java/org/apache/slider/core/launch/ClasspathConstructor.java @@ -123,12 +123,12 @@ public class ClasspathConstructor { public void addRemoteClasspathEnvVar() { -append(ApplicationConstants.Environment.CLASSPATH.$()); +append(ApplicationConstants.Environment.CLASSPATH.$$()); } public void insertRemoteClasspathEnvVar() { -append(ApplicationConstants.Environment.CLASSPATH.$()); +append(ApplicationConstants.Environment.CLASSPATH.$$()); } @@ -152,12 +152,22 @@ public class ClasspathConstructor { return dir; } + /** + * Split a classpath. This uses the local path separator so MUST NOT + * be used to work with remote classpaths + * @param localpath local path + * @return a splite + */ public Collection splitClasspath(String localpath) { String separator = System.getProperty("path.separator"); return StringUtils.getStringCollection(localpath, separator); } - public Collection javaVMClasspath() { + /** + * Get the local JVM classpath split up + * @return the list of entries on the JVM classpath env var + */ + public Collection localJVMClasspath() { return splitClasspath(System.getProperty("java.class.path")); }
git commit: SLIDER-199. Modify HBase package README to also add a manual way to create .zip packages
Repository: incubator-slider Updated Branches: refs/heads/develop e4723f5e5 -> 5bb80a42b SLIDER-199. Modify HBase package README to also add a manual way to create .zip packages Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/5bb80a42 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/5bb80a42 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/5bb80a42 Branch: refs/heads/develop Commit: 5bb80a42b1f5d6ea0c37d54b9dcb3f2ebf9251e1 Parents: e4723f5 Author: Sumit Mohanty Authored: Tue Jul 1 17:42:43 2014 -0700 Committer: Sumit Mohanty Committed: Tue Jul 1 17:42:48 2014 -0700 -- app-packages/hbase/README.txt | 54 +- pom.xml | 1 - 2 files changed, 36 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5bb80a42/app-packages/hbase/README.txt -- diff --git a/app-packages/hbase/README.txt b/app-packages/hbase/README.txt index a25ff8c..1d547a2 100644 --- a/app-packages/hbase/README.txt +++ b/app-packages/hbase/README.txt @@ -15,40 +15,58 @@ limitations under the License. --> -How to create a Slider package for HBase? +Create Slider App Package for HBase + +While appConfig.json and resources.json are not required for the package they +work well as the default configuration for Slider apps. So it is advisable that +when you create an application package for Slider, include sample/default +resources.json and appConfig.json for a minimal Yarn cluster. + +OPTION-I: Use mvn command +OPTION-II: Manual + +** OPTION - I (use mvn command) ** +You need the HBase version available on local maven repo to create the Slider App Package for HBase. The version of HBase used for the app package can be adjusted by adding a flag such as -Dhbase.version=0.98.3 -Copy the tarball for HBase: - cp ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz package/files/ +Download the tarball for HBase: + e.g. path to tarball ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz Use the following command to install HBase tarball locally: mvn install:install-file -Dfile= -DgroupId=org.apache.hbase -DartifactId=hbase -Dversion=0.98.3-hadoop2 -Dclassifier=bin -Dpackaging=tar.gz -After HBase tarball is publised locally in maven repository, you can use the following command: +After HBase tarball is published locally in maven repository, you can use the following command: mvn clean package -DskipTests -Phbase-app-package + App package can be found in app-packages/hbase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip -Create a zip package at the root of the package (/app-packages/hbase/) - zip -r hbase-v098.zip . - Verify the content using zip -Tv apache-slider-hbase-*.zip -While appConfig.json and resources.json are not required for the package they -work well as the default configuration for Slider apps. So it is advisable that -when you create an application package for Slider, include sample/default -resources.json and appConfig.json for a minimal Yarn cluster. +If an HBase version older than 0.98.3 is desired, it must be installed in the local maven repo. -If an HBase version older than 0.98.3 is desired, it must be installed in the -local maven repo. +A less descriptive file name can be specified with + -Dapp.package.name=HBase_98dot3 which would create a file HBase_98dot3.zip. -**Note that the LICENSE.txt and NOTICE.txt that are bundled with the app -package are designed for HBase 0.98.3 only and may need to be modified to be -applicable for other versions of the app package. +** OPTION - II (manual) ** +The Slider App Package for HBase can also be created manually. -A less descriptive file name can be specified with --Dapp.package.name=HBase_98dot3 which would create a file HBase_98dot3.zip. +Download the tarball for HBase: + e.g. path to tarball ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz + +Copy the hbase tarball to package/files + cp ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz package/files + +Edit appConfig.json/metainfo.xml + Replace 4 occurrences of "${hbase.version}" with the hbase version values such as "0.98.3-hadoop2" + Replace 1 occurrence of "${app.package.name}" with the desired app package name, e.g. "hbase-v098" + +Create a zip package at the root of the package (/app-packages/hbase/) + zip -r hbase-v098.zip . + +Verify the content using + zip -Tv hbase-v098.zip http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5bb80a42/pom.xml -- diff --git a/pom.xml b/pom.xml index e1f772b..460fb0d 100644 --- a/pom.xml +++ b/pom.xml @@ -39,7 +39,6 @@