svn commit: r948920 - /websites/production/slider/content/
Author: smohanty Date: Thu Apr 23 17:30:51 2015 New Revision: 948920 Log: Publishing svnmucc operation to slider site by smohanty Added: websites/production/slider/content/ - copied from r948919, websites/staging/slider/trunk/content/
svn commit: r1673115 - /incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
Author: smohanty
Date: Mon Apr 13 06:05:23 2015
New Revision: 1673115
URL: http://svn.apache.org/r1673115
Log:
Doc update for hello world app
Modified:
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
Modified:
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md?rev=1673115r1=1673114r2=1673115view=diff
==
---
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
(original)
+++
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
Mon Apr 13 06:05:23 2015
@@ -133,13 +133,10 @@ The config template has few mandatory pa
* `application.def` - location of the application definition package in
default FS (e.g. HDFS). *This is where the application package is stored.*
* `java_home` - location of java on target hosts
-* `package_list` - location of the app tarball in the app package (relative
path starting at **files**)
-* `config_types` - leave it as empty string
-* `site.global.app_user` - the user that is running the app (e.g. yarn)
-* `site.global.app_root` - location of app root, Slider extracts the tarball
at ${AGENT_WORK_ROOT}/app/install/, so append the relative path based on the
application tarball structure (e.g. jmemcached-1.0.0 for this package)
Add other parameters needed by the application itself. Currently we support
three parameters
+* `site.global.additional_cp` - this package uses this variable to provide the
location of other helper jars - ones found in the hadoop client jars location
(e.g. /usr/lib/hadoop/lib, your deployment may have the jars in a different
location)
* `site.global.xmx_val` - value of Xmx
* `site.global.xms_val` - value of Xms
* `site.global.memory_val` - value of memory
@@ -214,7 +211,6 @@ Note that the `parameter.py` file only r
process_cmd = format({java64_home}/bin/java -Xmx{xmx_val}
-Xms{xms_val} -classpath {app_root}/*:{additional_cp}
com.thimbleware.jmemcached.Main --memory={memory_val} --port={port})
Execute(process_cmd,
-user=params.app_user,
logoutput=False,
wait_for_finish=False,
pid_file=params.pid_file
@@ -228,10 +224,26 @@ Note that the `parameter.py` file only r
import params
env.set_params(params)
check_process_status(params.pid_file)
+
+ if __name__ == __main__:
+Memcached().execute()
Thats pretty much it. The script basically does the following:
* Expand the given tarball
* Reads the provided configuration and creates the command string
* Executes the command to start jmemcached
-* Start writes the PID into a file that is used to check status of the daemon
\ No newline at end of file
+* Start writes the PID into a file that is used to check status of the daemon
+
+## Debugging Tips
+End of the day, the above package runs the following command - post formatting
+
+format({java64_home}/bin/java -Xmx{xmx_val} -Xms{xms_val} -classpath
{app_root}/*:{additional_cp} com.thimbleware.jmemcached.Main
--memory={memory_val} --port={port})
+
+which, expands to
+
+/usr/jdk64/jdk1.7.0_67/bin/java -Xmx256m -Xms128m -classpath
/hadoop/yarn/local/usercache/yarn/appcache/application_1428879923172_0003/container_e01_1428879923172_0003_01_02/app/install/*:/usr/lib/hadoop/lib/*
com.thimbleware.jmemcached.Main --memory=200M --port=37539
+
+port and memory are based on the input provided. And, class path includes
YARN container location for the active container where the tarball got expanded.
+
+So, without Slider/YARN, one should be able to execute the command above and
see memcached up and running. It might be a better option to try the command
first to ensure that you have the right tarballs and environment to run
memcached.
\ No newline at end of file
svn commit: r947423 - /websites/production/slider/content/
Author: smohanty Date: Mon Apr 13 15:55:42 2015 New Revision: 947423 Log: Publishing svnmucc operation to slider site by smohanty Added: websites/production/slider/content/ - copied from r947422, websites/staging/slider/trunk/content/
[2/2] incubator-slider git commit: SLIDER-773. Add co-processor support for app packages (Thomas Liu via smohanty)
SLIDER-773. Add co-processor support for app packages (Thomas Liu via smohanty)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/016cd66a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/016cd66a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/016cd66a
Branch: refs/heads/develop
Commit: 016cd66aee97fe9752f78f9f8eb1283c162cc4d3
Parents: 4270267
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sat Apr 11 20:21:50 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sat Apr 11 20:22:10 2015 -0700
--
.../src/main/python/agent/ActionQueue.py| 13 +-
.../src/main/python/agent/AgentConfig.py| 2 +
.../src/main/python/agent/Controller.py | 7 +
.../python/agent/CustomServiceOrchestrator.py | 12 +-
slider-agent/src/main/python/agent/Heartbeat.py | 3 +-
slider-agent/src/main/python/agent/main.py | 3 +-
.../src/test/python/agent/TestHeartbeat.py | 6 +-
slider-core/pom.xml | 10 ++
.../core/persist/AppDefinitionPersister.java| 14 +-
.../providers/agent/AgentClientProvider.java| 4 +-
.../slider/providers/agent/AgentKeys.java | 2 +
.../providers/agent/AgentProviderService.java | 152 ---
.../slider/providers/agent/AgentUtils.java | 17 ++-
.../apache/slider/providers/agent/Command.java | 1 +
.../providers/agent/ComponentCommandOrder.java | 2 +
.../providers/agent/ComponentInstanceState.java | 132 +---
.../providers/agent/HeartbeatMonitor.java | 2 +
.../apache/slider/providers/agent/State.java| 7 +
.../application/metadata/AbstractComponent.java | 77 ++
.../metadata/AbstractMetainfoParser.java| 117 ++
.../metadata/AbstractMetainfoSchema.java| 72 +
.../metadata/AddoPackageMetainfoParser.java | 53 +++
.../agent/application/metadata/Application.java | 10 +-
.../metadata/ApplicationPackage.java| 51 +++
.../agent/application/metadata/Component.java | 19 +--
.../metadata/ComponentsInAddonPackage.java | 13 ++
.../agent/application/metadata/Metainfo.java| 27
.../application/metadata/MetainfoParser.java| 85 +--
.../web/rest/agent/ExecutionCommand.java| 11 ++
.../appmaster/web/rest/agent/HeartBeat.java | 12 ++
.../appmaster/web/rest/agent/Register.java | 9 ++
.../web/rest/agent/RegistrationResponse.java| 11 ++
.../agent/AgentMiniClusterTestBase.groovy | 25 +++
.../common/tools/TestWindowsSupport.groovy | 2 +-
.../agent/TestAgentProviderService.java | 41 +++--
.../TestApplicationWithAddonPackagesIT.groovy | 63
36 files changed, 911 insertions(+), 176 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/016cd66a/slider-agent/src/main/python/agent/ActionQueue.py
--
diff --git a/slider-agent/src/main/python/agent/ActionQueue.py
b/slider-agent/src/main/python/agent/ActionQueue.py
index 4cb5de7..6e3c3e2 100644
--- a/slider-agent/src/main/python/agent/ActionQueue.py
+++ b/slider-agent/src/main/python/agent/ActionQueue.py
@@ -63,10 +63,11 @@ class ActionQueue(threading.Thread):
self.controller = controller
self._stop = threading.Event()
self.tmpdir = config.getResolvedPath(AgentConfig.APP_TASK_DIR)
+self.componentPackage = ''
self.customServiceOrchestrator = CustomServiceOrchestrator(config,
controller,
self.queueOutAgentToggleLogger)
-
+
def stop(self):
self._stop.set()
@@ -121,6 +122,12 @@ class ActionQueue(threading.Thread):
'''
clusterName = command['clusterName']
commandId = command['commandId']
+if 'package' in command:
+self.componentPackage = command['package']
+else:
+self.componentPackage = ''
+
+logger.info(package received: + self.componentPackage + ;)
message = Executing command with id = {commandId} for role = {role} of \
cluster {cluster}.format(
@@ -155,7 +162,9 @@ class ActionQueue(threading.Thread):
if store_command:
logger.info(Component has indicated auto-restart. Saving details from
START command.)
-
+
+logger.info(will run: + str(command))
+
# running command
commandresult = self.customServiceOrchestrator.runCommand(command,
in_progress_status[
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/016cd66a/slider-agent/src/main/python/agent
[1/2] incubator-slider git commit: SLIDER-773. Add co-processor support for app packages (Thomas Liu via smohanty)
Repository: incubator-slider
Updated Branches:
refs/heads/develop 427026731 - 016cd66ae
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/016cd66a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
--
diff --git
a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
index f1d2c10..aa64ff6 100644
---
a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
+++
b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
@@ -98,6 +98,7 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
+import static org.easymock.EasyMock.anyBoolean;
import static org.easymock.EasyMock.anyObject;
import static org.easymock.EasyMock.createNiceMock;
import static org.easymock.EasyMock.expect;
@@ -303,7 +304,7 @@ public class TestAgentProviderService {
doReturn(access).when(mockAps).getAmState();
CommandScript cs = new CommandScript();
cs.setScript(scripts/hbase_master.py);
-doReturn(cs).when(mockAps).getScriptPathFromMetainfo(anyString());
+doReturn(cs).when(mockAps).getScriptPathForMasterPackage(anyString());
Metainfo metainfo = new Metainfo();
metainfo.setApplication(new Application());
@@ -311,7 +312,8 @@ public class TestAgentProviderService {
hm.setName(HBASE_MASTER);
metainfo.getApplication().addComponent(hm);
-
doReturn(metainfo).when(mockAps).getApplicationMetainfo(any(SliderFileSystem.class),
anyString());
+doReturn(metainfo).when(mockAps).getApplicationMetainfo(
+any(SliderFileSystem.class), Matchers.anyString(),
Matchers.anyBoolean());
Configuration conf = new Configuration();
conf.set(RegistryConstants.KEY_REGISTRY_ZK_ROOT,
@@ -325,7 +327,8 @@ public class TestAgentProviderService {
any(HeartBeatResponse.class),
eq(scripts/hbase_master.py),
eq((ComponentCommand)null),
- eq(600L));
+ eq(600L),
+ anyString());
doReturn(conf).when(mockAps).getConfig();
} catch (SliderException e) {
}
@@ -456,7 +459,7 @@ public class TestAgentProviderService {
CommandScript cs = new CommandScript();
cs.setScript(scripts/hbase_master.py);
doReturn(cs).when(mockAps)
-.getScriptPathFromMetainfo(anyString());
+.getScriptPathForMasterPackage(anyString());
Metainfo metainfo = new Metainfo();
Application application = new Application();
metainfo.setApplication(application);
@@ -471,7 +474,7 @@ public class TestAgentProviderService {
doNothing().when(mockAps).addInstallCommand(eq(HBASE_MASTER),
eq(mockcontainer_1), any(HeartBeatResponse.class),
eq(scripts/hbase_master.py), eq((ComponentCommand)null),
- eq(600L));
+ eq(600L), anyString());
doReturn(conf).when(mockAps).getConfig();
} catch (SliderException e) {
}
@@ -562,7 +565,8 @@ public class TestAgentProviderService {
any(HeartBeatResponse.class),
anyString(),
eq((ComponentCommand)null),
- Mockito.anyLong());
+ Mockito.anyLong(),
+ anyString());
doReturn(conf).when(mockAps).getConfig();
} catch (SliderException e) {
}
@@ -631,7 +635,8 @@ public class TestAgentProviderService {
any(HeartBeatResponse.class),
anyString(),
eq((ComponentCommand)null),
-
Mockito.anyLong());
+
Mockito.anyLong(),
+ null);
} catch (SliderException he) {
log.warn(he.getMessage());
} catch (IOException he) {
@@ -651,6 +656,8 @@ public class TestAgentProviderService {
application.addComponent(hbaseMaster);
metainfo.setApplication(application);
doReturn(metainfo).when(mockAps).getMetaInfo();
+doReturn(metainfo).when(mockAps).getApplicationMetainfo(
+any(SliderFileSystem.class), Matchers.anyString(),
Matchers.anyBoolean());
Register reg = new Register();
reg.setResponseId(0);
@@ -717,7 +724,7 @@ public class TestAgentProviderService {
application.addComponent(hbaseMaster);
metainfo.setApplication(application);
doReturn(metainfo).when(mockAps).getApplicationMetainfo(
-any(SliderFileSystem.class), anyString());
+any(SliderFileSystem.class), Matchers.anyString(),
Matchers.anyBoolean());
doReturn(metainfo).when(mockAps).getMetaInfo();
incubator-slider git commit: SLIDER-773. Add co-processor support for app packages (adding missing licence headers)
Repository: incubator-slider Updated Branches: refs/heads/develop 016cd66ae - 4197c8bb8 SLIDER-773. Add co-processor support for app packages (adding missing licence headers) Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/4197c8bb Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/4197c8bb Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/4197c8bb Branch: refs/heads/develop Commit: 4197c8bb8aef4490c0c4994bd8d22292b4b93dfe Parents: 016cd66 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sat Apr 11 22:31:20 2015 -0700 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sat Apr 11 22:31:20 2015 -0700 -- .../application/metadata/ApplicationPackage.java| 16 .../metadata/ComponentsInAddonPackage.java | 16 .../TestApplicationWithAddonPackagesIT.groovy | 16 3 files changed, 48 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4197c8bb/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ApplicationPackage.java -- diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ApplicationPackage.java b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ApplicationPackage.java index cce5a35..2ddcb71 100644 --- a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ApplicationPackage.java +++ b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ApplicationPackage.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the License); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.slider.providers.agent.application.metadata; import java.util.ArrayList; http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4197c8bb/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentsInAddonPackage.java -- diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentsInAddonPackage.java b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentsInAddonPackage.java index 9927400..3f62343 100644 --- a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentsInAddonPackage.java +++ b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentsInAddonPackage.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the License); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.slider.providers.agent.application.metadata; import org.apache.slider.core.exceptions.SliderException; http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/4197c8bb/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/TestApplicationWithAddonPackagesIT.groovy -- diff --git a/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/TestApplicationWithAddonPackagesIT.groovy
svn commit: r1667824 - in /incubator/slider/site/trunk/content/docs/slider_specs: index.md simple_pkg.md
Author: smohanty
Date: Thu Mar 19 18:20:56 2015
New Revision: 1667824
URL: http://svn.apache.org/r1667824
Log:
doc changes for packaging simplification
Added:
incubator/slider/site/trunk/content/docs/slider_specs/simple_pkg.md
Modified:
incubator/slider/site/trunk/content/docs/slider_specs/index.md
Modified: incubator/slider/site/trunk/content/docs/slider_specs/index.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/index.md?rev=1667824r1=1667823r2=1667824view=diff
==
--- incubator/slider/site/trunk/content/docs/slider_specs/index.md (original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/index.md Thu Mar 19
18:20:56 2015
@@ -38,7 +38,9 @@
##Specifications
-Refer to [Creating a Slider package for
Memcached](hello_world_slider_app.html) for a quick over view of how to write a
Slider app.
+Refer to [Creating a Slider package for
Memcached](hello_world_slider_app.html) for a quick over view of how to write a
Slider app.
+
+Packaging enhancements: [Simplified Packaging](simple_pkg.html) describes a
simplified version of packaging that Slider supports for applications that do
not need full capability of a Slider application package. *The work is
available in the develop branch and is targeted for the next relase.*
The entry points to leverage Slider are:
Added: incubator/slider/site/trunk/content/docs/slider_specs/simple_pkg.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/simple_pkg.md?rev=1667824view=auto
==
--- incubator/slider/site/trunk/content/docs/slider_specs/simple_pkg.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/simple_pkg.md Thu Mar
19 18:20:56 2015
@@ -0,0 +1,198 @@
+!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the License); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an AS IS BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--
+
+# Simplified Application Packaging
+
+An application may be a simple command as all of its dependencies are
pre-installed. In these cases creating a full application package zip file may
be too much. Similarly, applications artifacts can be organized in a folder
that the developer is continously editing while she is creating/debugging a
package. The need to zip up the package and uploading to FS explicitly can be
eliminated. The following simplifications are designed:
+
+* eliminate the need for a zip package
+* only mandate a metadata file (allow json structure as well)
+* allow reading an application definition from a local folder
+
+## Examples
+
+## A minimal application specified through metadata
+It is assummed that all binary dependencies are installed on the host machines.
+
+### Application specification
+**metainfo.json**
+
+{
+schemaVersion: 2.1,
+application: {
+name: MEMCACHED,
+components: [
+{
+name: MEMCACHED,
+commands: [
+{
+exec: /usr/jdk64/jdk1.7.0_67/bin/java -classpath
/usr/myapps/memcached/*:/usr/hadoop/lib/* com.thimbleware.jmemcached.Main
+}
+]
+ }
+]
+}
+}
+
+*At this point a resources.json is required but there is a plan to eliminate
it by introducing default resource requirements through slider-client.xml. A
common resources.json file is specified [here](#sample_resources)*
+
+### create
+
+slider create memcachedmin --metainfo metainfo.json --resources
resources.json
+
+## A minimal application with exports
+It is assummed that all binary dependencies are installed on the host
machines. Application asks Slider to allocate a dynamic port and then export it
to the registry. You can use the same syntax as specified in [specifying
exports](specifying_exports.html).
+
+### Application specification
+**metainfo.json**
+
+{
+schemaVersion: 2.1,
+application: {
+name: MEMCACHED,
+exportGroups: [
+{
+name: Servers,
+exports: [
+{
+name: host_port,
+value: ${MEMCACHED_HOST
svn commit: r944370 - /websites/production/slider/content/
Author: smohanty Date: Thu Mar 19 19:25:31 2015 New Revision: 944370 Log: Publishing svnmucc operation to slider site by smohanty Added: websites/production/slider/content/ - copied from r944369, websites/staging/slider/trunk/content/
[2/5] incubator-slider git commit: SLIDER-663. Make it easy to develop and deploy application packages that are essentially shell commands (part-I)
SLIDER-663. Make it easy to develop and deploy application packages that are
essentially shell commands (part-I)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/704e8136
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/704e8136
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/704e8136
Branch: refs/heads/develop
Commit: 704e81363a72444f0d27b6fe3ab43d46b7791231
Parents: 38fed99
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Mar 17 17:49:49 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Mar 17 17:49:49 2015 -0700
--
.../python/agent/CustomServiceOrchestrator.py | 56 -
.../python/scripts/shell_cmd/basic_installer.py | 55 +
.../src/main/python/scripts/shell_cmd/params.py | 29 +++
.../agent/TestCustomServiceOrchestrator.py | 104 -
.../org/apache/slider/api/InternalKeys.java | 8 +
.../org/apache/slider/client/SliderClient.java | 22 +-
.../org/apache/slider/common/SliderKeys.java| 12 +-
.../AbstractClusterBuildingActionArgs.java | 31 ++-
.../slider/common/params/AddonArgsDelegate.java | 54 +
.../apache/slider/common/params/Arguments.java | 3 +
.../slider/common/tools/CoreFileSystem.java | 27 ++-
.../apache/slider/common/tools/SliderUtils.java | 56 -
.../slider/core/build/InstanceBuilder.java | 6 +
.../core/persist/AppDefinitionPersister.java| 174 +++
.../slider/core/persist/InstancePaths.java | 4 +
.../providers/agent/AgentClientProvider.java| 57 +++--
.../slider/providers/agent/AgentKeys.java | 2 +
.../providers/agent/AgentProviderService.java | 211 ++-
.../slider/providers/agent/AgentUtils.java | 30 ++-
.../providers/agent/ComponentCommandOrder.java | 4 +-
.../agent/application/metadata/Application.java | 94 +++--
.../application/metadata/CommandOrder.java | 9 +-
.../application/metadata/CommandScript.java | 11 +-
.../agent/application/metadata/Component.java | 83 ++--
.../application/metadata/ComponentCommand.java | 85
.../agent/application/metadata/ConfigFile.java | 8 +-
.../agent/application/metadata/Export.java | 9 +-
.../agent/application/metadata/ExportGroup.java | 11 +-
.../agent/application/metadata/Metainfo.java| 24 ++-
.../application/metadata/MetainfoParser.java| 62 +-
.../agent/application/metadata/OSPackage.java | 9 +-
.../agent/application/metadata/OSSpecific.java | 11 +-
.../agent/application/metadata/Package.java | 60 ++
.../agent/application/metadata/Validate.java| 27 +++
.../web/rest/agent/ExecutionCommand.java| 18 +-
.../test_min_pkg/nc_ping_cmd/appConfig.json | 8 +
.../test_min_pkg/nc_ping_cmd/metainfo.json | 28 +++
.../test_min_pkg/nc_ping_cmd/resources.json | 17 ++
.../test_min_pkg/sleep_cmd/metainfo.json| 16 ++
.../test_min_pkg/sleep_cmd/resources.json | 17 ++
.../agent/TestAgentClientProvider2.java | 12 +-
.../agent/TestAgentProviderService.java | 129 +++-
.../metadata/TestMetainfoParser.java| 98 -
.../funtest/framework/CommandTestBase.groovy| 80 +++
.../funtest/lifecycle/AgentMinSleepIT.groovy| 110 ++
.../funtest/lifecycle/AgentPingSocketIT.groovy | 142 +
46 files changed, 1907 insertions(+), 216 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/704e8136/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
--
diff --git a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
index 119c926..87ce621 100644
--- a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
+++ b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
@@ -76,23 +76,37 @@ class CustomServiceOrchestrator():
override_output_files=True, store_command=False):
allocated_ports = {}
try:
+ py_file_list = []
+ json_path = None
+
script_type = command['commandParams']['script_type']
- script = command['commandParams']['script']
- timeout = int(command['commandParams']['command_timeout'])
task_id = command['taskId']
command_name = command['roleCommand']
- script_path = self.resolve_script_path(self.base_dir, script,
script_type)
- script_tuple = (script_path, self.base_dir)
-
tmpstrucoutfile = os.path.realpath(posixpath.join(self.tmp_dir,
structured-out-{0}.json.format(task_id)))
- if
[3/5] incubator-slider git commit: Merge branch 'develop' into feature/package_simplification_II
Merge branch 'develop' into feature/package_simplification_II
Conflicts:
slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/df352259
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/df352259
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/df352259
Branch: refs/heads/develop
Commit: df35225987ca7c9e521f64af49c1b3ed3c03af43
Parents: 704e813 96177ed
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Mar 17 21:08:32 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Mar 17 21:08:32 2015 -0700
--
.../hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll| Bin 92672 - 0 bytes
bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll | Bin 57344 - 0 bytes
.../hadoop-2.6.0-SNAPSHOT/bin/winutils.exe | Bin 108032 - 0 bytes
bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md | 22 ---
bin/windows/hadoop-2.6.0/bin/hadoop.dll | Bin 92672 - 0 bytes
bin/windows/hadoop-2.6.0/bin/hadoop.pdb | Bin 502784 - 0 bytes
bin/windows/hadoop-2.6.0/bin/hdfs.dll | Bin 57344 - 0 bytes
bin/windows/hadoop-2.6.0/bin/hdfs.pdb | Bin 347136 - 0 bytes
bin/windows/hadoop-2.6.0/bin/winutils.exe | Bin 108032 - 0 bytes
bin/windows/hadoop-2.6.0/bin/winutils.pdb | Bin 896000 - 0 bytes
.../providers/agent/AgentProviderService.java | 11 +-
.../web/rest/agent/ExecutionCommand.java| 10 +
12 files changed, 20 insertions(+), 23 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/df352259/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --cc
slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index 4eadf13,8f6f3bf..d2298f0
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@@ -1698,21 -1660,20 +1699,28 @@@ public class AgentProviderService exten
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
-MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
-cmd.setComponentConfigurations(componentConfigurations);
-
-cmd.setCommandParams(setCommandParameters(scriptPath, timeout, false));
+if(SliderUtils.isSet(scriptPath)) {
+ cmd.setCommandParams(commandParametersSet(scriptPath, timeout, false));
+} else {
+ // assume it to be default shell command
+ ComponentCommand effectiveCommand = compCmd;
+ if(effectiveCommand == null) {
+effectiveCommand =
ComponentCommand.getDefaultComponentCommand(INSTALL);
+ }
+ cmd.setCommandParams(commandParametersSet(effectiveCommand, timeout,
false));
+ configurations.get(global).put(exec_cmd,
effectiveCommand.getExec());
+}
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
+
response.addExecutionCommand(cmd);
+
+ log.debug(command looks like: + cmd.toString());
+ }
+
+ private MapString, MapString, String buildComponentConfigurations(
+ ConfTreeOperations appConf) {
+ return appConf.getComponents();
}
protected static String getPackageListFromApplication(Application
application) {
@@@ -1864,21 -1804,17 +1872,22 @@@
cmd.setRoleParams(roleParams);
cmd.getRoleParams().put(auto_restart,
Boolean.toString(isMarkedAutoRestart));
-cmd.setCommandParams(setCommandParameters(scriptPath, timeout, true));
-
-MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
-cmd.setComponentConfigurations(componentConfigurations);
-
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
-
cmd.setConfigurations(configurations);
+
+if(SliderUtils.isSet(scriptPath)) {
+ cmd.setCommandParams(commandParametersSet(scriptPath, timeout, true));
+} else {
+ if(startCommand == null) {
+throw new SliderException(Expected START command not found for
component + componentName);
+ }
+ cmd.setCommandParams(commandParametersSet(startCommand, timeout, true));
+ configurations.get(global).put(exec_cmd, startCommand.getExec());
+}
+
response.addExecutionCommand(cmd);
-
+
+ log.debug(command looks like: + cmd.toString());
// With start command, the corresponding
[1/5] incubator-slider git commit: SLIDER-663. Make it easy to develop and deploy application packages that are essentially shell commands (part-I)
Repository: incubator-slider
Updated Branches:
refs/heads/develop 3f07afbcb - 7f195f662
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/704e8136/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
index 418868c..3f1b7fe 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
@@ -19,6 +19,7 @@ package
org.apache.slider.providers.agent.application.metadata;
import org.apache.slider.common.tools.SliderUtils;
import org.apache.slider.core.exceptions.BadConfigException;
import org.apache.slider.core.exceptions.SliderException;
+import org.codehaus.jackson.annotate.JsonProperty;
import java.util.ArrayList;
import java.util.List;
@@ -26,21 +27,28 @@ import java.util.List;
/**
*
*/
-public class Component {
+public class Component implements Validate {
+
+ public static String TYPE_STANDARD = STANDARD;
+ public static String TYPE_DOCKER = DOCKER;
+ public static String CATEGORY_MASTER = MASTER;
+ public static String CATEGORY_SLAVE = SLAVE;
+ public static String CATEGORY_CLIENT = CLIENT;
+
String name;
- String category;
- String publishConfig;
- String minInstanceCount;
+ String category = CATEGORY_MASTER;
+ String publishConfig = Boolean.FALSE.toString();
+ String minInstanceCount = 0;
String maxInstanceCount;
- String autoStartOnFailure;
+ String autoStartOnFailure = Boolean.FALSE.toString();
String appExports;
String compExports;
CommandScript commandScript;
- ListComponentExport componentExports;
+ String type = TYPE_STANDARD;
+ ListComponentExport componentExports = new ArrayList();
+ ListComponentCommand commands = new ArrayList();
public Component() {
-publishConfig = Boolean.FALSE.toString();
-componentExports = new ArrayListComponentExport();
}
public String getName() {
@@ -51,6 +59,14 @@ public class Component {
this.name = name;
}
+ public String getType() {
+return type;
+ }
+
+ public void setType(String type) {
+this.type = type;
+ }
+
public String getCategory() {
return category;
}
@@ -95,6 +111,14 @@ public class Component {
return minInstanceCount;
}
+ public Boolean getAutoStartOnFailureBoolean() {
+if (SliderUtils.isUnset(getAutoStartOnFailure())) {
+ return Boolean.FALSE;
+}
+
+return Boolean.parseBoolean(getAutoStartOnFailure());
+ }
+
public int getMinInstanceCountInt() throws BadConfigException {
if (SliderUtils.isUnset(minInstanceCount)) {
return 0;
@@ -151,6 +175,11 @@ public class Component {
return Boolean.parseBoolean(this.autoStartOnFailure);
}
+ @JsonProperty(commands)
+ public ListComponentCommand getCommands() {
+return this.commands;
+ }
+
@Override
public String toString() {
final StringBuilder sb =
@@ -162,9 +191,39 @@ public class Component {
return sb.toString();
}
- class AutoRestartSettings {
-private boolean requiresAutoRestart;
-private int maxFailures;
-private int inThisManyMinutes;
+ public void validate(String version) throws SliderException {
+Metainfo.checkNonNull(getName(), name, component);
+Metainfo.checkNonNull(getCategory(), category, component);
+if (!getCategory().equals(CATEGORY_MASTER)
+ !getCategory().equals(CATEGORY_SLAVE)
+ !getCategory().equals(CATEGORY_CLIENT)) {
+ throw new SliderException(Invalid category for the component +
getCategory());
+}
+
+Metainfo.checkNonNull(getType(), type, component);
+if (!getType().equals(TYPE_DOCKER)
+ !getType().equals(TYPE_STANDARD)) {
+ throw new SliderException(Invalid type for the component + getType());
+}
+
+if (version.equals(Metainfo.VERSION_TWO_ZERO)) {
+ if (getType().equals(TYPE_DOCKER)) {
+throw new SliderException(TYPE_DOCKER + is not supported in version
+ Metainfo.VERSION_TWO_ZERO);
+ }
+
+ if (getCommands().size() 0) {
+throw new SliderException(commands are not supported in version +
Metainfo.VERSION_TWO_ZERO);
+ }
+}
+
+if (commandScript != null) {
+ commandScript.validate(version);
+}
+
+if (version.equals(Metainfo.VERSION_TWO_ONE)) {
+ for (ComponentCommand cc : getCommands()) {
+cc.validate(version);
+ }
+}
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/704e8136/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentCommand.java
--
diff
[5/5] incubator-slider git commit: Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop
Merge branch 'develop' of
https://git-wip-us.apache.org/repos/asf/incubator-slider into develop
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/7f195f66
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/7f195f66
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/7f195f66
Branch: refs/heads/develop
Commit: 7f195f66213d87f618d38e978baafc025db00a86
Parents: 210a1c9 3f07afb
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Mar 18 14:44:11 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Mar 18 14:44:11 2015 -0700
--
.../apache/slider/providers/agent/AgentProviderService.java| 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7f195f66/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --cc
slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index d2298f0,cc193f4..4b5f582
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@@ -1699,23 -1660,15 +1699,23 @@@ public class AgentProviderService exten
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
-MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
-cmd.setComponentConfigurations(componentConfigurations);
-
-cmd.setCommandParams(setCommandParameters(scriptPath, timeout, false));
+if(SliderUtils.isSet(scriptPath)) {
+ cmd.setCommandParams(commandParametersSet(scriptPath, timeout, false));
+} else {
+ // assume it to be default shell command
+ ComponentCommand effectiveCommand = compCmd;
+ if(effectiveCommand == null) {
+effectiveCommand =
ComponentCommand.getDefaultComponentCommand(INSTALL);
+ }
+ cmd.setCommandParams(commandParametersSet(effectiveCommand, timeout,
false));
+ configurations.get(global).put(exec_cmd,
effectiveCommand.getExec());
+}
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
+
response.addExecutionCommand(cmd);
- log.debug(command looks like: + cmd.toString());
+ log.debug(command looks like: {} , cmd);
}
private MapString, MapString, String buildComponentConfigurations(
@@@ -1726,21 -1679,14 +1726,21 @@@
protected static String getPackageListFromApplication(Application
application) {
String pkgFormatString = {\type\:\%s\,\name\:\%s\};
String pkgListFormatString = [%s];
- ListString packages = new ArrayList();
+ ListString packages = new ArrayList();
if (application != null) {
- ListOSSpecific osSpecifics = application.getOSSpecifics();
- if (osSpecifics != null osSpecifics.size() 0) {
-for (OSSpecific osSpecific : osSpecifics) {
- if (osSpecific.getOsType().equals(any)) {
-for (OSPackage osPackage : osSpecific.getPackages()) {
- packages.add(String.format(pkgFormatString,
osPackage.getType(), osPackage.getName()));
+ if (application.getPackages().size() 0) {
+ListPackage appPackages = application.getPackages();
+for (Package appPackage : appPackages) {
+ packages.add(String.format(pkgFormatString, appPackage.getType(),
appPackage.getName()));
+}
+ } else {
+ListOSSpecific osSpecifics = application.getOSSpecifics();
+if (osSpecifics != null osSpecifics.size() 0) {
+ for (OSSpecific osSpecific : osSpecifics) {
+if (osSpecific.getOsType().equals(any)) {
+ for (OSPackage osPackage : osSpecific.getPackages()) {
+packages.add(String.format(pkgFormatString,
osPackage.getType(), osPackage.getName()));
+ }
}
}
}
@@@ -1872,22 -1804,17 +1872,22 @@@
cmd.setRoleParams(roleParams);
cmd.getRoleParams().put(auto_restart,
Boolean.toString(isMarkedAutoRestart));
-cmd.setCommandParams(setCommandParameters(scriptPath, timeout, true));
-
-MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
-cmd.setComponentConfigurations(componentConfigurations);
-
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
-
cmd.setConfigurations(configurations);
+
+if(SliderUtils.isSet(scriptPath)) {
+
[4/5] incubator-slider git commit: SLIDER-663. Make it easy to develop and deploy application packages that are essentially shell commands (part-II)
SLIDER-663. Make it easy to develop and deploy application packages that are
essentially shell commands (part-II)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/210a1c94
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/210a1c94
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/210a1c94
Branch: refs/heads/develop
Commit: 210a1c94ca347380f7b22fba807c12dc1e30deff
Parents: df35225
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Mar 18 14:44:07 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Mar 18 14:44:07 2015 -0700
--
.../core/persist/AppDefinitionPersister.java| 64 +-
.../agent/TestAgentClientProvider2.java | 10 +-
.../agent/TestAppDefinitionPersister.java | 227 +++
3 files changed, 288 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/210a1c94/slider-core/src/main/java/org/apache/slider/core/persist/AppDefinitionPersister.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/core/persist/AppDefinitionPersister.java
b/slider-core/src/main/java/org/apache/slider/core/persist/AppDefinitionPersister.java
index 8f0f7b0..f394542 100644
---
a/slider-core/src/main/java/org/apache/slider/core/persist/AppDefinitionPersister.java
+++
b/slider-core/src/main/java/org/apache/slider/core/persist/AppDefinitionPersister.java
@@ -18,6 +18,7 @@
package org.apache.slider.core.persist;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.io.Files;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.fs.Path;
@@ -26,7 +27,6 @@ import
org.apache.slider.common.params.AbstractClusterBuildingActionArgs;
import org.apache.slider.common.tools.SliderFileSystem;
import org.apache.slider.common.tools.SliderUtils;
import org.apache.slider.core.conf.ConfTreeOperations;
-import org.apache.slider.core.exceptions.BadClusterStateException;
import org.apache.slider.core.exceptions.BadCommandArgumentsException;
import org.apache.slider.core.exceptions.BadConfigException;
import org.apache.slider.providers.agent.AgentKeys;
@@ -107,11 +107,23 @@ public class AppDefinitionPersister {
}
public void processSuppliedDefinitions(String clustername,
- AbstractClusterBuildingActionArgs
buildInfo,
- ConfTreeOperations appConf)
+ AbstractClusterBuildingActionArgs
buildInfo,
+ ConfTreeOperations appConf)
throws BadConfigException, IOException, BadCommandArgumentsException {
// if metainfo is provided add to the app instance
if (buildInfo.appMetaInfo != null) {
+
+ if (!buildInfo.appMetaInfo.canRead() || !buildInfo.appMetaInfo.isFile())
{
+throw new BadConfigException(--metainfo file cannot be read.);
+ }
+
+ if (buildInfo.appDef != null) {
+throw new BadConfigException(both --metainfo and --appdef may not be
specified.);
+ }
+ if
(SliderUtils.isSet(appConf.getGlobalOptions().get(AgentKeys.APP_DEF))) {
+throw new BadConfigException(application.def must not be set if
--metainfo is provided.);
+ }
+
File tempDir = Files.createTempDir();
File pkgSrcDir = new File(tempDir, default);
pkgSrcDir.mkdirs();
@@ -127,6 +139,14 @@ public class AppDefinitionPersister {
}
if (buildInfo.appDef != null) {
+ if
(SliderUtils.isSet(appConf.getGlobalOptions().get(AgentKeys.APP_DEF))) {
+throw new BadConfigException(application.def must not be set if
--appdef is provided.);
+ }
+
+ if (!buildInfo.appDef.exists()) {
+throw new BadConfigException(--appdef is not a valid path.);
+ }
+
Path appDirPath = sliderFileSystem.buildAppDefDirPath(clustername);
appDefinitions.add(new AppDefinition(appDirPath, buildInfo.appDef,
SliderKeys.DEFAULT_APP_PKG));
Path appDefPath = new Path(appDirPath, SliderKeys.DEFAULT_APP_PKG);
@@ -135,16 +155,30 @@ public class AppDefinitionPersister {
}
if (buildInfo.addonDelegate.getAddonMap().size() 0) {
+ if
(SliderUtils.isUnset(appConf.getGlobalOptions().get(AgentKeys.APP_DEF))) {
+throw new BadConfigException(addon package can only be specified if
main app package is specified.);
+ }
+
ListString addons = new ArrayListString();
MapString, String addonMap = buildInfo.addonDelegate.getAddonMap();
for (String key : addonMap.keySet()) {
+File defPath = new File(addonMap.get(key));
+if
[4/5] incubator-slider git commit: SLIDER-663. Make it easy to develop and deploy application packages that are essentially shell commands (part-I)
SLIDER-663. Make it easy to develop and deploy application packages that are
essentially shell commands (part-I)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/704e8136
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/704e8136
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/704e8136
Branch: refs/heads/feature/package_simplification_II
Commit: 704e81363a72444f0d27b6fe3ab43d46b7791231
Parents: 38fed99
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Mar 17 17:49:49 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Mar 17 17:49:49 2015 -0700
--
.../python/agent/CustomServiceOrchestrator.py | 56 -
.../python/scripts/shell_cmd/basic_installer.py | 55 +
.../src/main/python/scripts/shell_cmd/params.py | 29 +++
.../agent/TestCustomServiceOrchestrator.py | 104 -
.../org/apache/slider/api/InternalKeys.java | 8 +
.../org/apache/slider/client/SliderClient.java | 22 +-
.../org/apache/slider/common/SliderKeys.java| 12 +-
.../AbstractClusterBuildingActionArgs.java | 31 ++-
.../slider/common/params/AddonArgsDelegate.java | 54 +
.../apache/slider/common/params/Arguments.java | 3 +
.../slider/common/tools/CoreFileSystem.java | 27 ++-
.../apache/slider/common/tools/SliderUtils.java | 56 -
.../slider/core/build/InstanceBuilder.java | 6 +
.../core/persist/AppDefinitionPersister.java| 174 +++
.../slider/core/persist/InstancePaths.java | 4 +
.../providers/agent/AgentClientProvider.java| 57 +++--
.../slider/providers/agent/AgentKeys.java | 2 +
.../providers/agent/AgentProviderService.java | 211 ++-
.../slider/providers/agent/AgentUtils.java | 30 ++-
.../providers/agent/ComponentCommandOrder.java | 4 +-
.../agent/application/metadata/Application.java | 94 +++--
.../application/metadata/CommandOrder.java | 9 +-
.../application/metadata/CommandScript.java | 11 +-
.../agent/application/metadata/Component.java | 83 ++--
.../application/metadata/ComponentCommand.java | 85
.../agent/application/metadata/ConfigFile.java | 8 +-
.../agent/application/metadata/Export.java | 9 +-
.../agent/application/metadata/ExportGroup.java | 11 +-
.../agent/application/metadata/Metainfo.java| 24 ++-
.../application/metadata/MetainfoParser.java| 62 +-
.../agent/application/metadata/OSPackage.java | 9 +-
.../agent/application/metadata/OSSpecific.java | 11 +-
.../agent/application/metadata/Package.java | 60 ++
.../agent/application/metadata/Validate.java| 27 +++
.../web/rest/agent/ExecutionCommand.java| 18 +-
.../test_min_pkg/nc_ping_cmd/appConfig.json | 8 +
.../test_min_pkg/nc_ping_cmd/metainfo.json | 28 +++
.../test_min_pkg/nc_ping_cmd/resources.json | 17 ++
.../test_min_pkg/sleep_cmd/metainfo.json| 16 ++
.../test_min_pkg/sleep_cmd/resources.json | 17 ++
.../agent/TestAgentClientProvider2.java | 12 +-
.../agent/TestAgentProviderService.java | 129 +++-
.../metadata/TestMetainfoParser.java| 98 -
.../funtest/framework/CommandTestBase.groovy| 80 +++
.../funtest/lifecycle/AgentMinSleepIT.groovy| 110 ++
.../funtest/lifecycle/AgentPingSocketIT.groovy | 142 +
46 files changed, 1907 insertions(+), 216 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/704e8136/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
--
diff --git a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
index 119c926..87ce621 100644
--- a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
+++ b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
@@ -76,23 +76,37 @@ class CustomServiceOrchestrator():
override_output_files=True, store_command=False):
allocated_ports = {}
try:
+ py_file_list = []
+ json_path = None
+
script_type = command['commandParams']['script_type']
- script = command['commandParams']['script']
- timeout = int(command['commandParams']['command_timeout'])
task_id = command['taskId']
command_name = command['roleCommand']
- script_path = self.resolve_script_path(self.base_dir, script,
script_type)
- script_tuple = (script_path, self.base_dir)
-
tmpstrucoutfile = os.path.realpath(posixpath.join(self.tmp_dir,
[1/5] incubator-slider git commit: SLIDER-815 Binary files (dlls and exes) in git source and Slider source release (both .zip and .tar.gz)
Repository: incubator-slider Updated Branches: refs/heads/feature/package_simplification_II 38fed9935 - df3522598 SLIDER-815 Binary files (dlls and exes) in git source and Slider source release (both .zip and .tar.gz) Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/601ac135 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/601ac135 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/601ac135 Branch: refs/heads/feature/package_simplification_II Commit: 601ac135215aa183bfea9861ae7a62ec9688980d Parents: 38fed99 Author: Gour Saha gourks...@apache.org Authored: Mon Mar 16 11:02:56 2015 -0700 Committer: Gour Saha gourks...@apache.org Committed: Mon Mar 16 11:02:56 2015 -0700 -- .../hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll| Bin 92672 - 0 bytes bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll | Bin 57344 - 0 bytes .../hadoop-2.6.0-SNAPSHOT/bin/winutils.exe | Bin 108032 - 0 bytes bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md | 22 --- bin/windows/hadoop-2.6.0/bin/hadoop.dll | Bin 92672 - 0 bytes bin/windows/hadoop-2.6.0/bin/hadoop.pdb | Bin 502784 - 0 bytes bin/windows/hadoop-2.6.0/bin/hdfs.dll | Bin 57344 - 0 bytes bin/windows/hadoop-2.6.0/bin/hdfs.pdb | Bin 347136 - 0 bytes bin/windows/hadoop-2.6.0/bin/winutils.exe | Bin 108032 - 0 bytes bin/windows/hadoop-2.6.0/bin/winutils.pdb | Bin 896000 - 0 bytes 10 files changed, 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/601ac135/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll -- diff --git a/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll b/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll deleted file mode 100644 index 8da86d2..000 Binary files a/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/601ac135/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll -- diff --git a/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll b/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll deleted file mode 100644 index d96248c..000 Binary files a/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/601ac135/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/winutils.exe -- diff --git a/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/winutils.exe b/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/winutils.exe deleted file mode 100644 index a4edc85..000 Binary files a/bin/windows/hadoop-2.6.0-SNAPSHOT/bin/winutils.exe and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/601ac135/bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md -- diff --git a/bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md b/bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md deleted file mode 100644 index 9e4dd52..000 --- a/bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md +++ /dev/null @@ -1,22 +0,0 @@ -!--- - Licensed under the Apache License, Version 2.0 (the License); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an AS IS BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. --- - -# Windows Binaries - -This directory contains the native libraries needed for running Hadoop on -windows. It's here so that jenkins builds on windows do not -need to rely on a pre-installed version of hadoop. - -To ise - \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/601ac135/bin/windows/hadoop-2.6.0/bin/hadoop.dll -- diff --git a/bin/windows/hadoop-2.6.0/bin/hadoop.dll b/bin/windows/hadoop-2.6.0/bin/hadoop.dll deleted file mode 100644 index 78ad8a8..000 Binary files a/bin/windows/hadoop-2.6.0/bin/hadoop.dll and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/601ac135/bin/windows/hadoop-2.6.0/bin/hadoop.pdb -- diff --git a/bin/windows/hadoop-2.6.0/bin/hadoop.pdb b/bin/windows/hadoop-2.6.0/bin/hadoop.pdb deleted file mode 100644
[3/5] incubator-slider git commit: SLIDER-663. Make it easy to develop and deploy application packages that are essentially shell commands (part-I)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/704e8136/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
index 418868c..3f1b7fe 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
@@ -19,6 +19,7 @@ package
org.apache.slider.providers.agent.application.metadata;
import org.apache.slider.common.tools.SliderUtils;
import org.apache.slider.core.exceptions.BadConfigException;
import org.apache.slider.core.exceptions.SliderException;
+import org.codehaus.jackson.annotate.JsonProperty;
import java.util.ArrayList;
import java.util.List;
@@ -26,21 +27,28 @@ import java.util.List;
/**
*
*/
-public class Component {
+public class Component implements Validate {
+
+ public static String TYPE_STANDARD = STANDARD;
+ public static String TYPE_DOCKER = DOCKER;
+ public static String CATEGORY_MASTER = MASTER;
+ public static String CATEGORY_SLAVE = SLAVE;
+ public static String CATEGORY_CLIENT = CLIENT;
+
String name;
- String category;
- String publishConfig;
- String minInstanceCount;
+ String category = CATEGORY_MASTER;
+ String publishConfig = Boolean.FALSE.toString();
+ String minInstanceCount = 0;
String maxInstanceCount;
- String autoStartOnFailure;
+ String autoStartOnFailure = Boolean.FALSE.toString();
String appExports;
String compExports;
CommandScript commandScript;
- ListComponentExport componentExports;
+ String type = TYPE_STANDARD;
+ ListComponentExport componentExports = new ArrayList();
+ ListComponentCommand commands = new ArrayList();
public Component() {
-publishConfig = Boolean.FALSE.toString();
-componentExports = new ArrayListComponentExport();
}
public String getName() {
@@ -51,6 +59,14 @@ public class Component {
this.name = name;
}
+ public String getType() {
+return type;
+ }
+
+ public void setType(String type) {
+this.type = type;
+ }
+
public String getCategory() {
return category;
}
@@ -95,6 +111,14 @@ public class Component {
return minInstanceCount;
}
+ public Boolean getAutoStartOnFailureBoolean() {
+if (SliderUtils.isUnset(getAutoStartOnFailure())) {
+ return Boolean.FALSE;
+}
+
+return Boolean.parseBoolean(getAutoStartOnFailure());
+ }
+
public int getMinInstanceCountInt() throws BadConfigException {
if (SliderUtils.isUnset(minInstanceCount)) {
return 0;
@@ -151,6 +175,11 @@ public class Component {
return Boolean.parseBoolean(this.autoStartOnFailure);
}
+ @JsonProperty(commands)
+ public ListComponentCommand getCommands() {
+return this.commands;
+ }
+
@Override
public String toString() {
final StringBuilder sb =
@@ -162,9 +191,39 @@ public class Component {
return sb.toString();
}
- class AutoRestartSettings {
-private boolean requiresAutoRestart;
-private int maxFailures;
-private int inThisManyMinutes;
+ public void validate(String version) throws SliderException {
+Metainfo.checkNonNull(getName(), name, component);
+Metainfo.checkNonNull(getCategory(), category, component);
+if (!getCategory().equals(CATEGORY_MASTER)
+ !getCategory().equals(CATEGORY_SLAVE)
+ !getCategory().equals(CATEGORY_CLIENT)) {
+ throw new SliderException(Invalid category for the component +
getCategory());
+}
+
+Metainfo.checkNonNull(getType(), type, component);
+if (!getType().equals(TYPE_DOCKER)
+ !getType().equals(TYPE_STANDARD)) {
+ throw new SliderException(Invalid type for the component + getType());
+}
+
+if (version.equals(Metainfo.VERSION_TWO_ZERO)) {
+ if (getType().equals(TYPE_DOCKER)) {
+throw new SliderException(TYPE_DOCKER + is not supported in version
+ Metainfo.VERSION_TWO_ZERO);
+ }
+
+ if (getCommands().size() 0) {
+throw new SliderException(commands are not supported in version +
Metainfo.VERSION_TWO_ZERO);
+ }
+}
+
+if (commandScript != null) {
+ commandScript.validate(version);
+}
+
+if (version.equals(Metainfo.VERSION_TWO_ONE)) {
+ for (ComponentCommand cc : getCommands()) {
+cc.validate(version);
+ }
+}
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/704e8136/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/ComponentCommand.java
--
diff --git
[5/5] incubator-slider git commit: Merge branch 'develop' into feature/package_simplification_II
Merge branch 'develop' into feature/package_simplification_II
Conflicts:
slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/df352259
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/df352259
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/df352259
Branch: refs/heads/feature/package_simplification_II
Commit: df35225987ca7c9e521f64af49c1b3ed3c03af43
Parents: 704e813 96177ed
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Mar 17 21:08:32 2015 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Mar 17 21:08:32 2015 -0700
--
.../hadoop-2.6.0-SNAPSHOT/bin/hadoop.dll| Bin 92672 - 0 bytes
bin/windows/hadoop-2.6.0-SNAPSHOT/bin/hdfs.dll | Bin 57344 - 0 bytes
.../hadoop-2.6.0-SNAPSHOT/bin/winutils.exe | Bin 108032 - 0 bytes
bin/windows/hadoop-2.6.0-SNAPSHOT/readme.md | 22 ---
bin/windows/hadoop-2.6.0/bin/hadoop.dll | Bin 92672 - 0 bytes
bin/windows/hadoop-2.6.0/bin/hadoop.pdb | Bin 502784 - 0 bytes
bin/windows/hadoop-2.6.0/bin/hdfs.dll | Bin 57344 - 0 bytes
bin/windows/hadoop-2.6.0/bin/hdfs.pdb | Bin 347136 - 0 bytes
bin/windows/hadoop-2.6.0/bin/winutils.exe | Bin 108032 - 0 bytes
bin/windows/hadoop-2.6.0/bin/winutils.pdb | Bin 896000 - 0 bytes
.../providers/agent/AgentProviderService.java | 11 +-
.../web/rest/agent/ExecutionCommand.java| 10 +
12 files changed, 20 insertions(+), 23 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/df352259/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --cc
slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index 4eadf13,8f6f3bf..d2298f0
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@@ -1698,21 -1660,20 +1699,28 @@@ public class AgentProviderService exten
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
-MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
-cmd.setComponentConfigurations(componentConfigurations);
-
-cmd.setCommandParams(setCommandParameters(scriptPath, timeout, false));
+if(SliderUtils.isSet(scriptPath)) {
+ cmd.setCommandParams(commandParametersSet(scriptPath, timeout, false));
+} else {
+ // assume it to be default shell command
+ ComponentCommand effectiveCommand = compCmd;
+ if(effectiveCommand == null) {
+effectiveCommand =
ComponentCommand.getDefaultComponentCommand(INSTALL);
+ }
+ cmd.setCommandParams(commandParametersSet(effectiveCommand, timeout,
false));
+ configurations.get(global).put(exec_cmd,
effectiveCommand.getExec());
+}
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
+
response.addExecutionCommand(cmd);
+
+ log.debug(command looks like: + cmd.toString());
+ }
+
+ private MapString, MapString, String buildComponentConfigurations(
+ ConfTreeOperations appConf) {
+ return appConf.getComponents();
}
protected static String getPackageListFromApplication(Application
application) {
@@@ -1864,21 -1804,17 +1872,22 @@@
cmd.setRoleParams(roleParams);
cmd.getRoleParams().put(auto_restart,
Boolean.toString(isMarkedAutoRestart));
-cmd.setCommandParams(setCommandParameters(scriptPath, timeout, true));
-
-MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
-cmd.setComponentConfigurations(componentConfigurations);
-
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
-
cmd.setConfigurations(configurations);
+
+if(SliderUtils.isSet(scriptPath)) {
+ cmd.setCommandParams(commandParametersSet(scriptPath, timeout, true));
+} else {
+ if(startCommand == null) {
+throw new SliderException(Expected START command not found for
component + componentName);
+ }
+ cmd.setCommandParams(commandParametersSet(startCommand, timeout, true));
+ configurations.get(global).put(exec_cmd, startCommand.getExec());
+}
+
response.addExecutionCommand(cmd);
-
+
+ log.debug(command looks like: + cmd.toString());
// With start
[2/5] incubator-slider git commit: SLIDER-812 Making component configurations in appConfig available on the SliderAgent side (Thomas Liu via gourksaha)
SLIDER-812 Making component configurations in appConfig available on the
SliderAgent side (Thomas Liu via gourksaha)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/96177ed0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/96177ed0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/96177ed0
Branch: refs/heads/feature/package_simplification_II
Commit: 96177ed0c3822a96944d5c2296788937fdbcf51b
Parents: 601ac13
Author: Gour Saha gourks...@apache.org
Authored: Tue Mar 17 15:34:59 2015 -0700
Committer: Gour Saha gourks...@apache.org
Committed: Tue Mar 17 15:34:59 2015 -0700
--
.../providers/agent/AgentProviderService.java | 17 -
.../appmaster/web/rest/agent/ExecutionCommand.java | 10 ++
2 files changed, 26 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96177ed0/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index bdf168e..8f6f3bf 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -20,6 +20,7 @@ package org.apache.slider.providers.agent;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+
import org.apache.commons.io.FileUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
@@ -1659,10 +1660,20 @@ public class AgentProviderService extends
AbstractProviderService implements
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
+MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
+cmd.setComponentConfigurations(componentConfigurations);
+
cmd.setCommandParams(setCommandParameters(scriptPath, timeout, false));
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
response.addExecutionCommand(cmd);
+
+log.debug(command looks like: + cmd.toString());
+ }
+
+ private MapString, MapString, String buildComponentConfigurations(
+ ConfTreeOperations appConf) {
+return appConf.getComponents();
}
protected static String getPackageListFromApplication(Application
application) {
@@ -1795,11 +1806,15 @@ public class AgentProviderService extends
AbstractProviderService implements
cmd.setCommandParams(setCommandParameters(scriptPath, timeout, true));
+MapString, MapString, String componentConfigurations =
buildComponentConfigurations(appConf);
+cmd.setComponentConfigurations(componentConfigurations);
+
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
response.addExecutionCommand(cmd);
-
+
+log.debug(command looks like: + cmd.toString());
// With start command, the corresponding command for graceful stop needs to
// be sent. This will be used when a particular container is lost as per
RM,
// but then the agent is still running and heart-beating to the Slider AM.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96177ed0/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/ExecutionCommand.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/ExecutionCommand.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/ExecutionCommand.java
index 5fb3b5e..6144227 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/ExecutionCommand.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/ExecutionCommand.java
@@ -43,6 +43,7 @@ public class ExecutionCommand {
private MapString, String roleParams = null;
private String roleCommand;
private MapString, MapString, String configurations;
+ private MapString, MapString, String componentConfigurations;
private MapString, String commandParams;
private String serviceName;
private String componentName;
@@ -181,6 +182,15 @@ public class ExecutionCommand {
this.componentName = componentName;
}
+ public MapString, MapString, String getComponentConfigurations() {
+
[10/19] incubator-slider git commit: SLIDER-801 use AM cert generation for accumulo app package and its SSL funtests
SLIDER-801 use AM cert generation for accumulo app package and its SSL funtests
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/cdda804a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/cdda804a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/cdda804a
Branch: refs/heads/feature/package_simplification_II
Commit: cdda804a5043793e2b8acfdcf359e2a498926d91
Parents: bf293b9
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Tue Feb 24 07:13:11 2015 -0800
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Tue Mar 10 08:54:29 2015 -0700
--
app-packages/accumulo/appConfig-default.json| 1 -
.../accumulo/appConfig-secured-default.json | 1 -
.../accumulo/appConfig-ssl-default.json | 99 +++
.../accumulo/configuration/accumulo-env.xml | 2 +-
app-packages/accumulo/configuration/client.xml | 5 -
.../package/scripts/accumulo_configuration.py | 73 ++---
app-packages/accumulo/package/scripts/params.py | 21 +-
.../package/templates/accumulo-env.sh.j2| 42 ---
app-packages/accumulo/pom.xml | 1 +
app-packages/accumulo/src/assembly/accumulo.xml | 7 +
.../funtest/accumulo/AccumuloBasicIT.groovy | 6 +-
.../accumulo/AccumuloMonitorSSLIT.groovy| 8 +
.../funtest/accumulo/AccumuloReadWriteIT.groovy | 3 +-
.../accumulo/AccumuloReadWriteSSLIT.groovy | 27 +-
.../funtest/accumulo/AccumuloSSLTestBase.groovy | 86 +-
.../slider/funtest/accumulo/CertUtil.java | 275 ---
pom.xml | 2 +-
.../providers/agent/AgentProviderService.java | 6 +-
18 files changed, 176 insertions(+), 489 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/cdda804a/app-packages/accumulo/appConfig-default.json
--
diff --git a/app-packages/accumulo/appConfig-default.json
b/app-packages/accumulo/appConfig-default.json
index d3f963e..85c6985 100644
--- a/app-packages/accumulo/appConfig-default.json
+++ b/app-packages/accumulo/appConfig-default.json
@@ -28,7 +28,6 @@
site.proxy.port: ${ACCUMULO_PROXY.ALLOCATED_PORT}{PER_CONTAINER},
site.global.accumulo_root_password: NOT_USED,
-site.global.ssl_cert_dir: ssl,
site.global.monitor_protocol: http,
site.accumulo-site.instance.volumes: ${DEFAULT_DATA_DIR}/data,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/cdda804a/app-packages/accumulo/appConfig-secured-default.json
--
diff --git a/app-packages/accumulo/appConfig-secured-default.json
b/app-packages/accumulo/appConfig-secured-default.json
index 3332662..6d8abaa 100644
--- a/app-packages/accumulo/appConfig-secured-default.json
+++ b/app-packages/accumulo/appConfig-secured-default.json
@@ -29,7 +29,6 @@
site.proxy.port: ${ACCUMULO_PROXY.ALLOCATED_PORT}{PER_CONTAINER},
site.global.accumulo_root_password: NOT_USED,
-site.global.ssl_cert_dir: ssl,
site.global.monitor_protocol: http,
site.accumulo-site.instance.volumes: ${DEFAULT_DATA_DIR}/data,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/cdda804a/app-packages/accumulo/appConfig-ssl-default.json
--
diff --git a/app-packages/accumulo/appConfig-ssl-default.json
b/app-packages/accumulo/appConfig-ssl-default.json
new file mode 100644
index 000..9781e5f
--- /dev/null
+++ b/app-packages/accumulo/appConfig-ssl-default.json
@@ -0,0 +1,99 @@
+{
+ schema: http://example.org/specification/v2.0.0;,
+ metadata: {
+ },
+ global: {
+application.def: .slider/package/ACCUMULO/${app.package.name}.zip,
+java_home: ${app.java.home},
+
+slider.component.keystore.credential.alias.property:
rpc.javax.net.ssl.keyStorePassword,
+slider.component.truststore.credential.alias.property:
rpc.javax.net.ssl.trustStorePassword,
+
+site.global.app_root:
${AGENT_WORK_ROOT}/app/install/accumulo-${accumulo.version},
+site.global.app_user: ${app.user},
+site.global.user_group: ${app.user.group},
+
+site.global.metric_collector_host: ${NN_HOST},
+site.global.metric_collector_port: 6188,
+site.global.metric_collector_lib: ,
+
+site.accumulo-env.java_home: ${JAVA_HOME},
+site.accumulo-env.tserver_heapsize: 256m,
+site.accumulo-env.master_heapsize: 128m,
+site.accumulo-env.monitor_heapsize: 64m,
+site.accumulo-env.gc_heapsize: 64m,
+site.accumulo-env.other_heapsize: 128m,
+site.accumulo-env.hadoop_prefix: ${hadoop.dir},
+site.accumulo-env.hadoop_conf_dir: /etc/hadoop/conf,
+site.accumulo-env.zookeeper_home:
[19/19] incubator-slider git commit: minor cleanup of some classes while copying the code elsewhere
minor cleanup of some classes while copying the code elsewhere
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/38fed993
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/38fed993
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/38fed993
Branch: refs/heads/feature/package_simplification_II
Commit: 38fed99352d96f92a51813e54fab5e16123ca4b0
Parents: 3fbfca9
Author: Steve Loughran ste...@apache.org
Authored: Mon Mar 16 12:00:25 2015 +
Committer: Steve Loughran ste...@apache.org
Committed: Mon Mar 16 12:00:25 2015 +
--
.../main/java/org/apache/slider/api/SliderClusterProtocol.java | 1 -
.../main/java/org/apache/slider/common/tools/PortScanner.java| 2 +-
.../org/apache/slider/server/appmaster/rpc/SliderIPCService.java | 3 ++-
.../slider/server/appmaster/rpc/SliderRPCSecurityInfo.java | 4 +++-
4 files changed, 6 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38fed993/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
b/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
index dbb8c99..3250bb1 100644
--- a/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
+++ b/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
@@ -18,7 +18,6 @@
package org.apache.slider.api;
-import com.google.protobuf.RpcController;
import org.apache.hadoop.ipc.VersionedProtocol;
import org.apache.hadoop.security.KerberosInfo;
import org.apache.hadoop.yarn.exceptions.YarnException;
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38fed993/slider-core/src/main/java/org/apache/slider/common/tools/PortScanner.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/PortScanner.java
b/slider-core/src/main/java/org/apache/slider/common/tools/PortScanner.java
index 60fd008..b5b21ce 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/PortScanner.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/PortScanner.java
@@ -69,7 +69,7 @@ public class PortScanner {
}
public int getAvailablePort() throws SliderException {
-if (remainingPortsToCheck!=null) {
+if (remainingPortsToCheck != null) {
return getAvailablePortViaPortArray();
} else {
return getAvailablePortViaCounter();
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38fed993/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderIPCService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderIPCService.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderIPCService.java
index a924940..d5822e9 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderIPCService.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderIPCService.java
@@ -108,7 +108,8 @@ public class SliderIPCService extends AbstractService
CertificateManager certificateManager,
StateAccessForProviders state,
QueueAccess actionQueues,
- MetricsAndMonitoring metricsAndMonitoring, ContentCache cache) {
+ MetricsAndMonitoring metricsAndMonitoring,
+ ContentCache cache) {
super(SliderIPCService);
Preconditions.checkArgument(amOperations != null, null amOperations);
Preconditions.checkArgument(state != null, null appState);
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/38fed993/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderRPCSecurityInfo.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderRPCSecurityInfo.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderRPCSecurityInfo.java
index 215db1a..4fd4910 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderRPCSecurityInfo.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/rpc/SliderRPCSecurityInfo.java
@@ -30,7 +30,9 @@ import org.apache.slider.common.SliderXmlConfKeys;
import java.lang.annotation.Annotation;
/**
- * This is where security may go
+ * This is where security information goes.
+ * It is referred to in the
codeMETA-INF/services/org.apache.hadoop.security.SecurityInfo/code
+ * resource of this JAR, which is used to find the
[03/19] incubator-slider git commit: SLIDER-792 fix cert store file name collisions
SLIDER-792 fix cert store file name collisions
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/14aa6edb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/14aa6edb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/14aa6edb
Branch: refs/heads/feature/package_simplification_II
Commit: 14aa6edbdf677cb5fd91d0195030238ebeb25e42
Parents: de41cb5
Author: Jon Maron jma...@hortonworks.com
Authored: Tue Mar 3 11:06:14 2015 -0800
Committer: Jon Maron jma...@hortonworks.com
Committed: Tue Mar 3 11:06:14 2015 -0800
--
.../providers/agent/AgentProviderService.java | 24 ++--
.../services/security/CertificateManager.java | 19 +--
.../services/security/KeystoreGenerator.java| 8 +-
.../server/services/security/SecurityStore.java | 66 ++
.../security/SecurityStoreGenerator.java| 8 +-
.../services/security/StoresGenerator.java | 21 ++--
.../services/security/TruststoreGenerator.java | 9 +-
.../security/TestCertificateManager.java| 123 +--
8 files changed, 200 insertions(+), 78 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/14aa6edb/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index 74eda97..1eeb8e5 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -95,6 +95,7 @@ import
org.apache.slider.server.appmaster.web.rest.agent.RegistrationResponse;
import org.apache.slider.server.appmaster.web.rest.agent.RegistrationStatus;
import org.apache.slider.server.appmaster.web.rest.agent.StatusCommand;
import org.apache.slider.server.services.security.CertificateManager;
+import org.apache.slider.server.services.security.SecurityStore;
import org.apache.slider.server.services.security.StoresGenerator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -468,20 +469,21 @@ public class AgentProviderService extends
AbstractProviderService implements
MapOperations compOps = instanceDefinition.getAppConfOperations()
.getComponent(role);
// generate and localize security stores
-File[] stores = generateSecurityStores(container, role,
- instanceDefinition, compOps);
-for (File store : stores) {
+SecurityStore[] stores = generateSecurityStores(container, role,
+instanceDefinition,
compOps);
+for (SecurityStore store : stores) {
LocalResource keystoreResource = fileSystem.createAmResource(
- uploadSecurityResource(store, fileSystem), LocalResourceType.FILE);
- launcher.addLocalResource(String.format(secstores/%s.p12, role),
+ uploadSecurityResource(store.getFile(), fileSystem),
LocalResourceType.FILE);
+ launcher.addLocalResource(String.format(secstores/%s-%s.p12,
+ store.getType(), role),
keystoreResource);
}
}
- private File[] generateSecurityStores(Container container,
- String role,
- AggregateConf instanceDefinition,
- MapOperations compOps)
+ private SecurityStore[] generateSecurityStores(Container container,
+ String role,
+ AggregateConf
instanceDefinition,
+ MapOperations compOps)
throws SliderException, IOException {
return
StoresGenerator.generateSecurityStores(container.getNodeId().getHost(),
container.getId().toString(), role,
@@ -489,8 +491,8 @@ public class AgentProviderService extends
AbstractProviderService implements
}
private boolean areStoresRequested(MapOperations compOps) {
-return compOps != null ? Boolean.valueOf(compOps.
-getOptionBool(SliderKeys.COMP_STORES_REQUIRED_KEY, false)) : false;
+return compOps != null ? compOps.
+getOptionBool(SliderKeys.COMP_STORES_REQUIRED_KEY, false) : false;
}
private void localizeContainerSSLResources(ContainerLauncher launcher,
[13/19] incubator-slider git commit: SLIDER-319 -block out ganglia integration
SLIDER-319 -block out ganglia integration
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/148a4354
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/148a4354
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/148a4354
Branch: refs/heads/feature/package_simplification_II
Commit: 148a4354d7fe04b71c527fc8d81968741b27ee65
Parents: e6c988f
Author: Steve Loughran ste...@apache.org
Authored: Thu Mar 12 15:54:53 2015 -0700
Committer: Steve Loughran ste...@apache.org
Committed: Thu Mar 12 15:54:53 2015 -0700
--
slider-core/pom.xml | 2 ++
.../management/MetricsBindingService.java | 18 ++
2 files changed, 12 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/148a4354/slider-core/pom.xml
--
diff --git a/slider-core/pom.xml b/slider-core/pom.xml
index 0633446..22bc868 100644
--- a/slider-core/pom.xml
+++ b/slider-core/pom.xml
@@ -343,10 +343,12 @@
artifactIdmetrics-core/artifactId
/dependency
+!--
dependency
groupIdcom.codahale.metrics/groupId
artifactIdmetrics-ganglia/artifactId
/dependency
+--
dependency
groupIdcom.codahale.metrics/groupId
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/148a4354/slider-core/src/main/java/org/apache/slider/server/appmaster/management/MetricsBindingService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/management/MetricsBindingService.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/management/MetricsBindingService.java
index fb1dbd2..f8646bf 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/management/MetricsBindingService.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/management/MetricsBindingService.java
@@ -22,9 +22,7 @@ import com.codahale.metrics.JmxReporter;
import com.codahale.metrics.MetricRegistry;
import com.codahale.metrics.ScheduledReporter;
import com.codahale.metrics.Slf4jReporter;
-import com.codahale.metrics.ganglia.GangliaReporter;
import com.google.common.base.Preconditions;
-import info.ganglia.gmetric4j.gmetric.GMetric;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.service.CompositeService;
import org.apache.slider.server.services.workflow.ClosingService;
@@ -86,23 +84,26 @@ public class MetricsBindingService extends CompositeService
// Ganglia
if (conf.getBoolean(METRICS_GANGLIA_ENABLED, false)) {
- GangliaReporter gangliaReporter;
+ log.warn(Ganglia integration is not implemented);
+/*
+ // This is all disabled due to transitive dependencies on an LGPL library
+ com.codahale.metrics.ganglia.GangliaReporter gangliaReporter;
String host = conf.getTrimmed(METRICS_GANGLIA_HOST, );
int port = conf.getInt(METRICS_GANGLIA_PORT, DEFAULT_GANGLIA_PORT);
int interval = conf.getInt(METRICS_GANGLIA_REPORT_INTERVAL, 60);
int ttl = 1;
- GMetric.UDPAddressingMode
- mcast = GMetric.UDPAddressingMode.getModeForAddress(host);
+ info.ganglia.gmetric4j.gmetric.GMetric.UDPAddressingMode
+ mcast =
info.ganglia.gmetric4j.gmetric.GMetric.UDPAddressingMode.getModeForAddress(host);
boolean ganglia31 = conf.getBoolean(METRICS_GANGLIA_VERSION_31, true);
- final GMetric ganglia =
- new GMetric(
+ final info.ganglia.gmetric4j.gmetric.GMetric ganglia =
+ new info.ganglia.gmetric4j.gmetric.GMetric(
host,
port,
mcast,
ttl,
ganglia31);
- gangliaReporter = GangliaReporter.forRegistry(metrics)
+ gangliaReporter =
com.codahale.metrics.ganglia.GangliaReporter.forRegistry(metrics)
.convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(
TimeUnit.MILLISECONDS)
@@ -111,6 +112,7 @@ public class MetricsBindingService extends CompositeService
addService(new ClosingServiceScheduledReporter(gangliaReporter));
summary.append(String.format(, Ganglia at %s:%d interval=%d,
host, port, interval));
+ */
}
// Logging
[18/19] incubator-slider git commit: SLIDER-801 fix newline bug in config copy in accumulo client script
SLIDER-801 fix newline bug in config copy in accumulo client script Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/3fbfca9e Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/3fbfca9e Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/3fbfca9e Branch: refs/heads/feature/package_simplification_II Commit: 3fbfca9e629bd2252a4c8736c97b701e3d5313df Parents: 833c2bb Author: Billie Rinaldi billie.rina...@gmail.com Authored: Fri Mar 13 13:29:35 2015 -0700 Committer: Billie Rinaldi billie.rina...@gmail.com Committed: Fri Mar 13 13:29:35 2015 -0700 -- app-packages/accumulo/package/scripts/accumulo_client.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/3fbfca9e/app-packages/accumulo/package/scripts/accumulo_client.py -- diff --git a/app-packages/accumulo/package/scripts/accumulo_client.py b/app-packages/accumulo/package/scripts/accumulo_client.py index 31ef684..cd11269 100644 --- a/app-packages/accumulo/package/scripts/accumulo_client.py +++ b/app-packages/accumulo/package/scripts/accumulo_client.py @@ -62,7 +62,7 @@ class AccumuloClient(Script): for line in content: index = line.find(=) if index 0: -configs[line[0:index]] = line[index+1:] +configs[line[0:index]] = line[index+1:].rstrip() if 'instance.rpc.ssl.enabled' in configs and configs['instance.rpc.ssl.enabled']=='true': Logger.info(Configuring client SSL) self.check_provider_contains(client_params.credential_provider,
[11/19] incubator-slider git commit: SLIDER-769 Modify Slider app packages for seamless integration with AMS provider (Ambari Metrics Service)
SLIDER-769 Modify Slider app packages for seamless integration with AMS
provider (Ambari Metrics Service)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/9a7bedf2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/9a7bedf2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/9a7bedf2
Branch: refs/heads/feature/package_simplification_II
Commit: 9a7bedf294d3db554bd3ec9fe8a93ffd6d28f83d
Parents: cdda804
Author: Gour Saha gourks...@apache.org
Authored: Tue Mar 10 14:01:22 2015 -0700
Committer: Gour Saha gourks...@apache.org
Committed: Tue Mar 10 14:01:22 2015 -0700
--
.../package/templates/hadoop-metrics2-accumulo.properties.j2 | 2 +-
.../templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2 | 2 +-
.../templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2 | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9a7bedf2/app-packages/accumulo/package/templates/hadoop-metrics2-accumulo.properties.j2
--
diff --git
a/app-packages/accumulo/package/templates/hadoop-metrics2-accumulo.properties.j2
b/app-packages/accumulo/package/templates/hadoop-metrics2-accumulo.properties.j2
index 900a507..8b31ef5 100644
---
a/app-packages/accumulo/package/templates/hadoop-metrics2-accumulo.properties.j2
+++
b/app-packages/accumulo/package/templates/hadoop-metrics2-accumulo.properties.j2
@@ -34,7 +34,7 @@
rpc.collector={{metric_collector_host}}:{{metric_collector_port}}
accumulo.sink.timeline.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
accumulo.sink.timeline.period=10
accumulo.sink.timeline.collector={{metric_collector_host}}:{{metric_collector_port}}
-accumulo.sink.timeline.serviceName-prefix={{app_name}}
+accumulo.sink.timeline.serviceName={{app_name}}
{% else %}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9a7bedf2/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
--
diff --git
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
index 65816a2..f56b211 100644
---
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
+++
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
@@ -46,7 +46,7 @@
rpc.collector={{metric_collector_host}}:{{metric_collector_port}}
hbase.sink.timeline.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
hbase.sink.timeline.period=10
hbase.sink.timeline.collector={{metric_collector_host}}:{{metric_collector_port}}
-hbase.sink.timeline.serviceName-prefix={{app_name}}
+hbase.sink.timeline.serviceName={{app_name}}
{% else %}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9a7bedf2/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
--
diff --git
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
index 65816a2..f56b211 100644
---
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
+++
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
@@ -46,7 +46,7 @@
rpc.collector={{metric_collector_host}}:{{metric_collector_port}}
hbase.sink.timeline.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
hbase.sink.timeline.period=10
hbase.sink.timeline.collector={{metric_collector_host}}:{{metric_collector_port}}
-hbase.sink.timeline.serviceName-prefix={{app_name}}
+hbase.sink.timeline.serviceName={{app_name}}
{% else %}
[09/19] incubator-slider git commit: Merge branch 'feature/SLIDER-802-cert-sign' into develop
Merge branch 'feature/SLIDER-802-cert-sign' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/bf293b9d Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/bf293b9d Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/bf293b9d Branch: refs/heads/feature/package_simplification_II Commit: bf293b9df7dd06ee80a6c8876d0463da38e61a8f Parents: 670bfb3 c1b8346 Author: Jon Maron jma...@hortonworks.com Authored: Fri Mar 6 18:14:46 2015 -0800 Committer: Jon Maron jma...@hortonworks.com Committed: Fri Mar 6 18:14:46 2015 -0800 -- .../slider/api/SliderClusterProtocol.java |5 +- .../org/apache/slider/api/proto/Messages.java | 1503 +- .../slider/api/proto/RestTypeMarshalling.java | 24 + .../slider/api/proto/SliderClusterAPI.java | 81 +- .../org/apache/slider/client/SliderClient.java | 83 +- .../apache/slider/client/SliderClientAPI.java |2 +- .../client/ipc/SliderClusterOperations.java | 21 +- .../slider/common/params/ActionClientArgs.java | 24 + .../apache/slider/common/params/Arguments.java |5 + .../slider/common/params/SliderActions.java |4 +- .../server/appmaster/SliderAppMaster.java |6 +- .../rpc/SliderClusterProtocolPBImpl.java| 12 + .../rpc/SliderClusterProtocolProxy.java | 10 + .../server/appmaster/rpc/SliderIPCService.java | 36 + .../services/security/CertificateManager.java | 28 +- .../src/main/proto/SliderClusterMessages.proto | 11 + .../src/main/proto/SliderClusterProtocol.proto |5 +- .../agent/TestAgentClientProvider2.java |7 - .../AMClientCertStoreRetrievalIT.groovy | 174 ++ 19 files changed, 2003 insertions(+), 38 deletions(-) --
[16/19] incubator-slider git commit: SLIDER-763 add app name option to client command
SLIDER-763 add app name option to client command
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/d9b26524
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/d9b26524
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/d9b26524
Branch: refs/heads/feature/package_simplification_II
Commit: d9b265249218122b0df6b8064faef422f8e21158
Parents: dd2a70b
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Fri Mar 13 09:55:14 2015 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Fri Mar 13 10:36:46 2015 -0700
--
slider-assembly/src/main/scripts/slider | 4 ++-
.../org/apache/slider/client/SliderClient.java | 3 ++-
.../providers/AbstractClientProvider.java | 8 +++---
.../providers/agent/AgentClientProvider.java| 25 +-
.../agent/TestAgentClientProvider2.java | 27 +---
5 files changed, 53 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d9b26524/slider-assembly/src/main/scripts/slider
--
diff --git a/slider-assembly/src/main/scripts/slider
b/slider-assembly/src/main/scripts/slider
index 97dabaf..28dc205 100755
--- a/slider-assembly/src/main/scripts/slider
+++ b/slider-assembly/src/main/scripts/slider
@@ -41,5 +41,7 @@ cmd = [pyExecPath + + currDir + /slider.py]
cmd.extend(args)
finalCmd = .join(cmd)
-result = subprocess.call(finalCmd, shell=True)
+env = os.environ.copy()
+env['SLIDER_HOME'] = os.path.dirname(currDir)
+result = subprocess.call(finalCmd, env=env, shell=True)
sys.exit(result)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d9b26524/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index df3ba95..84ed5a4 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -1067,7 +1067,8 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
INSTALL,
clientInfo.installLocation,
pkgFile,
-config);
+config,
+clientInfo.name);
return EXIT_SUCCESS;
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d9b26524/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
b/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
index b5aafd4..1a1baf7 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
@@ -224,16 +224,18 @@ public abstract class AbstractClientProvider extends
Configured {
* Process client operations for applications such as install, configure
* @param fileSystem
* @param operation
- * @param config
- * @param clientPackage
* @param clientInstallPath
+ * @param clientPackage
+ * @param config
+ * @param name
* @throws SliderException
*/
public void processClientOperation(SliderFileSystem fileSystem,
String operation,
File clientInstallPath,
File clientPackage,
- JSONObject config)
+ JSONObject config,
+ String name)
throws SliderException {
throw new SliderException(Provider does not support client operations.);
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d9b26524/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
index 94d4c97..9da269d 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
+++
[14/19] incubator-slider git commit: SLIDER-811 added provider/alias options to client getcertstore command
SLIDER-811 added provider/alias options to client getcertstore command
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/dd2a70bd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/dd2a70bd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/dd2a70bd
Branch: refs/heads/feature/package_simplification_II
Commit: dd2a70bd63a60178de8a5d67e5ab2a43ac281420
Parents: 148a435
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Fri Mar 13 06:47:24 2015 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Fri Mar 13 10:36:46 2015 -0700
--
.../org/apache/slider/client/SliderClient.java | 50 +--
.../slider/common/params/ActionClientArgs.java | 11 +-
.../apache/slider/common/params/Arguments.java | 1 +
.../AMClientCertStoreRetrievalIT.groovy | 131 ++-
4 files changed, 148 insertions(+), 45 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/dd2a70bd/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index 8c1a9b2..df3ba95 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -697,7 +697,7 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
return startCluster(clustername, createArgs);
}
- private void checkForCredentials(Configuration conf,
+ private static void checkForCredentials(Configuration conf,
ConfTree tree) throws IOException {
if (tree.credentials == null || tree.credentials.size()==0) {
log.info(No credentials requested);
@@ -726,9 +726,6 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
br = new BufferedReader(new InputStreamReader(System.in));
}
char[] pass = readPassword(alias, br);
-if (pass == null)
- throw new IOException(Could not read credentials for + alias +
- from stdin);
credentialProvider.createCredentialEntry(alias, pass);
credentialProvider.flush();
Arrays.fill(pass, ' ');
@@ -742,9 +739,21 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
}
}
+ private static char[] readOnePassword(String alias) throws IOException {
+BufferedReader br = null;
+try {
+ br = new BufferedReader(new InputStreamReader(System.in));
+ return readPassword(alias, br);
+} finally {
+ if (br != null) {
+br.close();
+ }
+}
+ }
+
// using a normal reader instead of a secure one,
// because stdin is not hooked up to the command line
- private char[] readPassword(String alias, BufferedReader br)
+ private static char[] readPassword(String alias, BufferedReader br)
throws IOException {
char[] cred = null;
@@ -763,6 +772,9 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
}
if (newPassword2 != null) Arrays.fill(newPassword2, ' ');
} while (noMatch);
+if (cred == null)
+ throw new IOException(Could not read credentials for + alias +
+ from stdin);
return cred;
}
@@ -944,7 +956,7 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
}
if (clientInfo.name == null) {
- throw new BadCommandArgumentsException(No applicaiton name specified\n
+ throw new BadCommandArgumentsException(No application name specified\n
+ CommonArgs.usage(serviceArgs,
ACTION_CLIENT));
}
@@ -975,14 +987,26 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
String password = clientInfo.password;
if (password == null) {
- // get a password
- BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
- try {
-password = String.valueOf(readPassword(type.name(), br));
- } finally {
-if (br != null) {
- br.close();
+ String provider = clientInfo.provider;
+ String alias = clientInfo.alias;
+ if (provider != null alias != null) {
+Configuration conf = new Configuration(getConfig());
+conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH, provider);
+char[] chars = conf.getPassword(alias);
+if (chars == null) {
+ CredentialProvider
[01/19] incubator-slider git commit: SLIDER-769 Modify Slider app packages for seamless integration with AMS provider (Ambari Metrics Service)
Repository: incubator-slider
Updated Branches:
refs/heads/feature/package_simplification_II de41cb5bd - 38fed9935
SLIDER-769 Modify Slider app packages for seamless integration with AMS
provider (Ambari Metrics Service)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/54bcc86c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/54bcc86c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/54bcc86c
Branch: refs/heads/feature/package_simplification_II
Commit: 54bcc86c5ab0bb6a56f107f57378e7fde4a59f13
Parents: de41cb5
Author: Gour Saha gourks...@apache.org
Authored: Tue Mar 3 00:26:53 2015 -0800
Committer: Gour Saha gourks...@apache.org
Committed: Tue Mar 3 00:26:53 2015 -0800
--
.../templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2 | 2 +-
.../templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2 | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/54bcc86c/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
--
diff --git
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
index 43aea01..65816a2 100644
---
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
+++
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
@@ -46,7 +46,7 @@
rpc.collector={{metric_collector_host}}:{{metric_collector_port}}
hbase.sink.timeline.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
hbase.sink.timeline.period=10
hbase.sink.timeline.collector={{metric_collector_host}}:{{metric_collector_port}}
-hbase.sink.timeline.serviceName-prefix={{app_name}}-master
+hbase.sink.timeline.serviceName-prefix={{app_name}}
{% else %}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/54bcc86c/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
--
diff --git
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
index 2ec4b24..65816a2 100644
---
a/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
+++
b/app-packages/hbase/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
@@ -46,7 +46,7 @@
rpc.collector={{metric_collector_host}}:{{metric_collector_port}}
hbase.sink.timeline.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
hbase.sink.timeline.period=10
hbase.sink.timeline.collector={{metric_collector_host}}:{{metric_collector_port}}
-hbase.sink.timeline.serviceName-prefix={{app_name}}-rs
+hbase.sink.timeline.serviceName-prefix={{app_name}}
{% else %}
[15/19] incubator-slider git commit: SLIDER-801 make client install command get SSL certs from AM when SSL is enabled
SLIDER-801 make client install command get SSL certs from AM when SSL is enabled
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/42f1fbdc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/42f1fbdc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/42f1fbdc
Branch: refs/heads/feature/package_simplification_II
Commit: 42f1fbdc7011a08a70a41d3c7b6f2d0cc4229203
Parents: d9b2652
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Fri Mar 13 10:24:22 2015 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Fri Mar 13 10:36:46 2015 -0700
--
.../accumulo/clientInstallConfig-default.json | 3 +-
.../accumulo/package/scripts/accumulo_client.py | 48 +-
.../accumulo/package/scripts/client_params.py | 13 +-
app-packages/accumulo/pom.xml | 3 +-
.../funtest/accumulo/AccumuloScriptIT.groovy| 9 +-
.../funtest/accumulo/AccumuloScriptSSLIT.groovy | 147 +++
.../resources/clientInstallConfig-ssl-test.json | 7 +
.../resources/clientInstallConfig-test.json | 8 -
.../src/test/resources/test_shell_cmd_file | 20 +++
9 files changed, 240 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/42f1fbdc/app-packages/accumulo/clientInstallConfig-default.json
--
diff --git a/app-packages/accumulo/clientInstallConfig-default.json
b/app-packages/accumulo/clientInstallConfig-default.json
index 02176dc..db01ac4 100644
--- a/app-packages/accumulo/clientInstallConfig-default.json
+++ b/app-packages/accumulo/clientInstallConfig-default.json
@@ -1,7 +1,6 @@
{
schema:http://example.org/specification/v2.0.0;,
global:{
-client_root: {app_install_dir}/accumulo-${accumulo.version},
-slider_home_dir: ${slider.dir}
+client_root: {app_install_dir}/accumulo-${accumulo.version}
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/42f1fbdc/app-packages/accumulo/package/scripts/accumulo_client.py
--
diff --git a/app-packages/accumulo/package/scripts/accumulo_client.py
b/app-packages/accumulo/package/scripts/accumulo_client.py
index 2da02dc..31ef684 100644
--- a/app-packages/accumulo/package/scripts/accumulo_client.py
+++ b/app-packages/accumulo/package/scripts/accumulo_client.py
@@ -21,12 +21,19 @@ limitations under the License.
from resource_management import *
class AccumuloClient(Script):
+ def check_provider_contains(self, provider, alias):
+try:
+ Execute( format(hadoop credential list -provider {provider} |
+ grep -i {alias}))
+except:
+ raise Fail(format({provider} did not contain {alias}, try running
+'hadoop credential create {alias} -provider
+{provider}' or configure SSL certs manually))
+
def install(self, env):
import client_params
env.set_params(client_params)
self.install_packages(env)
-Directory(client_params.conf_dir,
- content=format({conf_dir}/templates))
jarname = SliderAccumuloUtils.jar
File(format({client_root}/lib/{jarname}),
mode=0644,
@@ -44,8 +51,45 @@ class AccumuloClient(Script):
mode=0755
)
if client_params.app_name:
+ Logger.info(Creating configs for app %s % client_params.app_name)
+ Directory(client_params.conf_dir,
+content=format({conf_dir}/templates))
Execute( format({bin_dir}/accumulo-slider
--appconf {client_root}/conf --app {app_name} getconf
))
+ configs = {}
+ with open(format({client_root}/conf/client.conf),r) as fp:
+content = fp.readlines()
+for line in content:
+ index = line.find(=)
+ if index 0:
+configs[line[0:index]] = line[index+1:]
+ if 'instance.rpc.ssl.enabled' in configs and
configs['instance.rpc.ssl.enabled']=='true':
+Logger.info(Configuring client SSL)
+self.check_provider_contains(client_params.credential_provider,
+ client_params.keystore_alias)
+self.check_provider_contains(client_params.credential_provider,
+ client_params.truststore_alias)
+configs['general.security.credential.provider.paths'] =
client_params.credential_provider
+configs['rpc.javax.net.ssl.keyStore'] = client_params.keystore_path
+configs['rpc.javax.net.ssl.keyStoreType'] = client_params.store_type
+configs['rpc.javax.net.ssl.trustStore'] = client_params.truststore_path
+configs['rpc.javax.net.ssl.trustStoreType'] =
[05/19] incubator-slider git commit: SLIDER-756 revise accumulo-slider script to work with slider client --install command
SLIDER-756 revise accumulo-slider script to work with slider client --install
command
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/670bfb32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/670bfb32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/670bfb32
Branch: refs/heads/feature/package_simplification_II
Commit: 670bfb32fd26769fd5f08b4609f05fb6c915da45
Parents: 909b7c1
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Wed Mar 4 23:14:24 2015 -0800
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Wed Mar 4 23:14:24 2015 -0800
--
app-packages/accumulo/accumulo-slider | 53 ---
app-packages/accumulo/accumulo-slider.py| 378 ---
.../accumulo/clientInstallConfig-default.json | 7 +
app-packages/accumulo/metainfo.xml | 2 +-
.../accumulo/package/files/accumulo-slider | 59 +++
.../accumulo/package/files/accumulo-slider.py | 182 +
.../accumulo/package/scripts/accumulo_client.py | 37 +-
.../accumulo/package/scripts/client_params.py | 39 ++
.../package/templates/accumulo-slider-env.sh.j2 | 17 +
app-packages/accumulo/pom.xml | 14 +-
app-packages/accumulo/src/assembly/accumulo.xml | 8 +-
.../funtest/accumulo/AccumuloScriptIT.groovy| 91 +++--
.../funtest/accumulo/AccumuloSliderShell.groovy | 7 -
.../resources/clientInstallConfig-test.json | 8 +
14 files changed, 415 insertions(+), 487 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/670bfb32/app-packages/accumulo/accumulo-slider
--
diff --git a/app-packages/accumulo/accumulo-slider
b/app-packages/accumulo/accumulo-slider
deleted file mode 100644
index 8f9cb06..000
--- a/app-packages/accumulo/accumulo-slider
+++ /dev/null
@@ -1,53 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# License); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an AS IS BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Resolve links - $0 may be a softlink
-PRG=${0}
-
-while [ -h ${PRG} ]; do
- ls=`ls -ld ${PRG}`
- link=`expr $ls : '.*- \(.*\)$'`
- if expr $link : '/.*' /dev/null; then
-PRG=$link
- else
-PRG=`dirname ${PRG}`/$link
- fi
-done
-
-# find python = 2.6
-if [ -a /usr/bin/python2.6 ]; then
- PYTHON=/usr/bin/python2.6
-fi
-
-if [ -z $PYTHON ]; then
- PYTHON=/usr/bin/python
-fi
-
-# check for version
-majversion=`$PYTHON -V 21 | awk '{print $2}' | cut -d'.' -f1`
-minversion=`$PYTHON -V 21 | awk '{print $2}' | cut -d'.' -f2`
-numversion=$(( 10 * $majversion + $minversion))
-if (( $numversion 26 )); then
- echo Need python version 2.6
- exit 1
-fi
-
-ACCUMULO_SLIDER_BIN_DIR=`dirname ${PRG}`
-
-$PYTHON ${ACCUMULO_SLIDER_BIN_DIR}/accumulo-slider.py $@
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/670bfb32/app-packages/accumulo/accumulo-slider.py
--
diff --git a/app-packages/accumulo/accumulo-slider.py
b/app-packages/accumulo/accumulo-slider.py
deleted file mode 100644
index 1eccf59..000
--- a/app-packages/accumulo/accumulo-slider.py
+++ /dev/null
@@ -1,378 +0,0 @@
-#!/usr/bin/python
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# License); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an AS IS BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-import sys
-import json
-import glob
-import tempfile
[07/19] incubator-slider git commit: SLIDER-802 add client keystore and trustore retrieval commands
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/c1b83467/slider-core/src/main/java/org/apache/slider/api/proto/SliderClusterAPI.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/api/proto/SliderClusterAPI.java
b/slider-core/src/main/java/org/apache/slider/api/proto/SliderClusterAPI.java
index 0d473fc..3441625 100644
---
a/slider-core/src/main/java/org/apache/slider/api/proto/SliderClusterAPI.java
+++
b/slider-core/src/main/java/org/apache/slider/api/proto/SliderClusterAPI.java
@@ -274,12 +274,12 @@ public final class SliderClusterAPI {
com.google.protobuf.RpcCallbackorg.apache.slider.api.proto.Messages.WrappedJsonProto
done);
/**
- * coderpc
signCertificate(.org.apache.slider.api.SignCertificateRequestProto) returns
(.org.apache.slider.api.SignCertificateResponseProto);/code
+ * coderpc
getClientCertificateStore(.org.apache.slider.api.GetCertificateStoreRequestProto)
returns (.org.apache.slider.api.GetCertificateStoreResponseProto);/code
*/
- public abstract void signCertificate(
+ public abstract void getClientCertificateStore(
com.google.protobuf.RpcController controller,
- org.apache.slider.api.proto.Messages.SignCertificateRequestProto
request,
-
com.google.protobuf.RpcCallbackorg.apache.slider.api.proto.Messages.SignCertificateResponseProto
done);
+ org.apache.slider.api.proto.Messages.GetCertificateStoreRequestProto
request,
+
com.google.protobuf.RpcCallbackorg.apache.slider.api.proto.Messages.GetCertificateStoreResponseProto
done);
}
@@ -463,11 +463,11 @@ public final class SliderClusterAPI {
}
@java.lang.Override
-public void signCertificate(
+public void getClientCertificateStore(
com.google.protobuf.RpcController controller,
-org.apache.slider.api.proto.Messages.SignCertificateRequestProto
request,
-
com.google.protobuf.RpcCallbackorg.apache.slider.api.proto.Messages.SignCertificateResponseProto
done) {
- impl.signCertificate(controller, request, done);
+
org.apache.slider.api.proto.Messages.GetCertificateStoreRequestProto request,
+
com.google.protobuf.RpcCallbackorg.apache.slider.api.proto.Messages.GetCertificateStoreResponseProto
done) {
+ impl.getClientCertificateStore(controller, request, done);
}
};
@@ -537,7 +537,7 @@ public final class SliderClusterAPI {
case 21:
return impl.getLiveResources(controller,
(org.apache.slider.api.proto.Messages.EmptyPayloadProto)request);
case 22:
- return impl.signCertificate(controller,
(org.apache.slider.api.proto.Messages.SignCertificateRequestProto)request);
+ return impl.getClientCertificateStore(controller,
(org.apache.slider.api.proto.Messages.GetCertificateStoreRequestProto)request);
default:
throw new java.lang.AssertionError(Can't get here.);
}
@@ -597,7 +597,7 @@ public final class SliderClusterAPI {
case 21:
return
org.apache.slider.api.proto.Messages.EmptyPayloadProto.getDefaultInstance();
case 22:
- return
org.apache.slider.api.proto.Messages.SignCertificateRequestProto.getDefaultInstance();
+ return
org.apache.slider.api.proto.Messages.GetCertificateStoreRequestProto.getDefaultInstance();
default:
throw new java.lang.AssertionError(Can't get here.);
}
@@ -657,7 +657,7 @@ public final class SliderClusterAPI {
case 21:
return
org.apache.slider.api.proto.Messages.WrappedJsonProto.getDefaultInstance();
case 22:
- return
org.apache.slider.api.proto.Messages.SignCertificateResponseProto.getDefaultInstance();
+ return
org.apache.slider.api.proto.Messages.GetCertificateStoreResponseProto.getDefaultInstance();
default:
throw new java.lang.AssertionError(Can't get here.);
}
@@ -919,12 +919,12 @@ public final class SliderClusterAPI {
com.google.protobuf.RpcCallbackorg.apache.slider.api.proto.Messages.WrappedJsonProto
done);
/**
- * coderpc
signCertificate(.org.apache.slider.api.SignCertificateRequestProto) returns
(.org.apache.slider.api.SignCertificateResponseProto);/code
+ * coderpc
getClientCertificateStore(.org.apache.slider.api.GetCertificateStoreRequestProto)
returns (.org.apache.slider.api.GetCertificateStoreResponseProto);/code
*/
-public abstract void signCertificate(
+public abstract void getClientCertificateStore(
com.google.protobuf.RpcController controller,
-org.apache.slider.api.proto.Messages.SignCertificateRequestProto
request,
-
[08/19] incubator-slider git commit: SLIDER-802 add client keystore and trustore retrieval commands
SLIDER-802 add client keystore and trustore retrieval commands
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/c1b83467
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/c1b83467
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/c1b83467
Branch: refs/heads/feature/package_simplification_II
Commit: c1b83467cebc90b4e258d963ac8c5d90b85f3b1c
Parents: b18daa3
Author: Jon Maron jma...@hortonworks.com
Authored: Fri Mar 6 18:14:13 2015 -0800
Committer: Jon Maron jma...@hortonworks.com
Committed: Fri Mar 6 18:14:13 2015 -0800
--
.../slider/api/SliderClusterProtocol.java |5 +-
.../org/apache/slider/api/proto/Messages.java | 1188 +++---
.../slider/api/proto/RestTypeMarshalling.java | 24 +
.../slider/api/proto/SliderClusterAPI.java | 75 +-
.../org/apache/slider/client/SliderClient.java | 83 +-
.../apache/slider/client/SliderClientAPI.java |2 +-
.../client/ipc/SliderClusterOperations.java | 29 +-
.../slider/common/params/ActionClientArgs.java | 24 +
.../apache/slider/common/params/Arguments.java |5 +
.../slider/common/params/SliderActions.java |4 +-
.../server/appmaster/SliderAppMaster.java |6 +-
.../rpc/SliderClusterProtocolPBImpl.java|9 +-
.../rpc/SliderClusterProtocolProxy.java |4 +-
.../server/appmaster/rpc/SliderIPCService.java | 36 +-
.../services/security/CertificateManager.java | 28 +-
.../src/main/proto/SliderClusterMessages.proto | 15 +-
.../src/main/proto/SliderClusterProtocol.proto |4 +-
.../agent/TestAgentClientProvider2.java |7 -
.../AMClientCertStoreRetrievalIT.groovy | 174 +++
19 files changed, 902 insertions(+), 820 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/c1b83467/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
b/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
index b4e91fc..dbb8c99 100644
--- a/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
+++ b/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
@@ -161,7 +161,6 @@ public interface SliderClusterProtocol extends
VersionedProtocol {
Messages.WrappedJsonProto getLiveResources(Messages.EmptyPayloadProto
request) throws IOException;
- Messages.SignCertificateResponseProto signCertificate(
- Messages.SignCertificateRequestProto request) throws IOException;
-
+ Messages.GetCertificateStoreResponseProto
getClientCertificateStore(Messages.GetCertificateStoreRequestProto request)
+ throws IOException;
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/c1b83467/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
b/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
index 3e1cad3..53c7d45 100644
--- a/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
+++ b/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
@@ -25129,73 +25129,88 @@ public final class Messages {
//
@@protoc_insertion_point(class_scope:org.apache.slider.api.WrappedJsonProto)
}
- public interface SignCertificateRequestProtoOrBuilder
+ public interface GetCertificateStoreRequestProtoOrBuilder
extends com.google.protobuf.MessageOrBuilder {
-// required string hostname = 1;
+// optional string hostname = 1;
/**
- * coderequired string hostname = 1;/code
+ * codeoptional string hostname = 1;/code
*/
boolean hasHostname();
/**
- * coderequired string hostname = 1;/code
+ * codeoptional string hostname = 1;/code
*/
java.lang.String getHostname();
/**
- * coderequired string hostname = 1;/code
+ * codeoptional string hostname = 1;/code
*/
com.google.protobuf.ByteString
getHostnameBytes();
-// required string certRequest = 2;
+// required string requesterId = 2;
/**
- * coderequired string certRequest = 2;/code
+ * coderequired string requesterId = 2;/code
*/
-boolean hasCertRequest();
+boolean hasRequesterId();
/**
- * coderequired string certRequest = 2;/code
+ * coderequired string requesterId = 2;/code
*/
-java.lang.String getCertRequest();
+java.lang.String getRequesterId();
/**
- * coderequired string certRequest = 2;/code
+ * coderequired
[17/19] incubator-slider git commit: SLIDER-811 clean up existing provider in cert store IT
SLIDER-811 clean up existing provider in cert store IT
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/833c2bb7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/833c2bb7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/833c2bb7
Branch: refs/heads/feature/package_simplification_II
Commit: 833c2bb7db1af99173f1ff9898bb54b759b4c96a
Parents: 42f1fbd
Author: Billie Rinaldi billie.rina...@gmail.com
Authored: Fri Mar 13 12:00:54 2015 -0700
Committer: Billie Rinaldi billie.rina...@gmail.com
Committed: Fri Mar 13 12:00:54 2015 -0700
--
.../slider/funtest/lifecycle/AMClientCertStoreRetrievalIT.groovy | 3 +++
1 file changed, 3 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/833c2bb7/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/AMClientCertStoreRetrievalIT.groovy
--
diff --git
a/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/AMClientCertStoreRetrievalIT.groovy
b/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/AMClientCertStoreRetrievalIT.groovy
index 3163387..dae4cc3 100644
---
a/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/AMClientCertStoreRetrievalIT.groovy
+++
b/slider-funtest/src/test/groovy/org/apache/slider/funtest/lifecycle/AMClientCertStoreRetrievalIT.groovy
@@ -148,6 +148,9 @@ implements FuntestProperties, Arguments, SliderExitCodes,
SliderActions {
UserGroupInformation.getCurrentUser().getShortUserName() + /test- +
APPLICATION_NAME + .jceks
Path providerPath = ProviderUtils.unnestUri(new URI(providerString))
+if (clusterFS.exists(providerPath)) {
+ clusterFS.delete(providerPath, false)
+}
Configuration conf = loadSliderConf()
conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
providerString)
[12/19] incubator-slider git commit: SLIDER-806 HBase App Package build issues
SLIDER-806 HBase App Package build issues Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e6c988f5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e6c988f5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e6c988f5 Branch: refs/heads/feature/package_simplification_II Commit: e6c988f55f192699d71a6d01b2391a162ef3138a Parents: 9a7bedf Author: Gour Saha gourks...@apache.org Authored: Wed Mar 11 07:47:23 2015 -0700 Committer: Gour Saha gourks...@apache.org Committed: Wed Mar 11 07:47:23 2015 -0700 -- app-packages/hbase-win/README.txt | 10 -- app-packages/hbase/README.md | 11 +-- 2 files changed, 17 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e6c988f5/app-packages/hbase-win/README.txt -- diff --git a/app-packages/hbase-win/README.txt b/app-packages/hbase-win/README.txt index 6389fb2..8e9c3b2 100644 --- a/app-packages/hbase-win/README.txt +++ b/app-packages/hbase-win/README.txt @@ -21,17 +21,23 @@ appConfig-default.json and resources-default.json are not required to be package These files are included as reference configuration for Slider apps and are suitable for a one-node cluster. - To create the app package you will need the HBase tarball and invoke mvn command with appropriate parameters. +Note: Post 0.94 versions of hbase, tarball filenames have a -bin suffix (e.g. + hbase-0.98.10.1-hadoop2-bin.tar.gz), although the untarred top level + directory does not (e.g hbase-0.98.10.1-hadoop2). We suggest renaming + the tar ball file to remove the -bin suffix before proceeding (e.g. + hbase-0.98.10.1-hadoop2.tar.gz). In fact the mvn command below assumes + you have done so, or else app create will fail. + Command: mvn clean package -Phbase-app-package-win -Dpkg.version=version -Dpkg.name=file name of app zip file -Dpkg.src=folder location where the pkg is available Example: mvn clean package -Phbase-app-package-win -Dpkg.version=0.98.5-hadoop2 - -Dpkg.name=hbase-0.98.5-hadoop2-bin.zip + -Dpkg.name=hbase-0.98.5-hadoop2.zip -Dpkg.src=/Users/user1/Downloads App package can be found in http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e6c988f5/app-packages/hbase/README.md -- diff --git a/app-packages/hbase/README.md b/app-packages/hbase/README.md index ae9763b..51a7ade 100644 --- a/app-packages/hbase/README.md +++ b/app-packages/hbase/README.md @@ -24,6 +24,13 @@ for a one-node cluster. OPTION-I: Use a downloaded tarball OPTION-II: Use the tarball from the local repository +Note: Post 0.94 versions of hbase, tarball filenames have a -bin suffix (e.g. + hbase-0.98.10.1-hadoop2-bin.tar.gz), although the untarred top level + directory does not (e.g hbase-0.98.10.1-hadoop2). We suggest renaming + the tar ball file to remove the -bin suffix before proceeding with either + of the below options (e.g. hbase-0.98.10.1-hadoop2.tar.gz). In fact all the + mvn commands below assumes you have done so, or else app create will fail. + ## OPTION - I To create the app package you will need the HBase tarball and invoke mvn command @@ -37,7 +44,7 @@ Command: Example: mvn clean package -Phbase-app-package -Dpkg.version=0.98.5-hadoop2 - -Dhbase.version=hbase-0.98.5-hadoop2-bin.tar.gz + -Dhbase.version=hbase-0.98.5-hadoop2.tar.gz -Dpkg.src=/Users/user1/Downloads/0.98.5-hadoop2 App package can be found in @@ -49,7 +56,7 @@ App package can be found in You need the HBase version available on local maven repo to create the Slider App Package for HBase. Download the tarball for HBase: - e.g. path to tarball `~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz` + e.g. path to tarball `~/Downloads/hbase-0.98.3-hadoop2.tar.gz` Note for Phoenix user: If you want Phoenix to be deployed, you should repackage HBase tarball such that phoenix-server.jar is contained in the lib directory of the tarball.
[04/19] incubator-slider git commit: Merge branch 'feature/SLIDER-792_avoid_cert_store_name_collision' into develop
Merge branch 'feature/SLIDER-792_avoid_cert_store_name_collision' into develop Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/909b7c1a Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/909b7c1a Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/909b7c1a Branch: refs/heads/feature/package_simplification_II Commit: 909b7c1ae34522f3b83f634fba96f836d7083320 Parents: 24622cb 14aa6ed Author: Jon Maron jma...@hortonworks.com Authored: Tue Mar 3 11:06:28 2015 -0800 Committer: Jon Maron jma...@hortonworks.com Committed: Tue Mar 3 11:06:28 2015 -0800 -- .../providers/agent/AgentProviderService.java | 24 ++-- .../services/security/CertificateManager.java | 19 +-- .../services/security/KeystoreGenerator.java| 8 +- .../server/services/security/SecurityStore.java | 66 ++ .../security/SecurityStoreGenerator.java| 8 +- .../services/security/StoresGenerator.java | 21 ++-- .../services/security/TruststoreGenerator.java | 9 +- .../security/TestCertificateManager.java| 123 +-- 8 files changed, 200 insertions(+), 78 deletions(-) --
[06/19] incubator-slider git commit: SLIDER-802 stub protobuf call and proxy operations to sign certificates
SLIDER-802 stub protobuf call and proxy operations to sign certificates
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b18daa35
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b18daa35
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b18daa35
Branch: refs/heads/feature/package_simplification_II
Commit: b18daa35d105e76d9739f5aaab035c828872fa24
Parents: 909b7c1
Author: Steve Loughran ste...@apache.org
Authored: Thu Mar 5 14:33:29 2015 -0800
Committer: Steve Loughran ste...@apache.org
Committed: Thu Mar 5 14:33:29 2015 -0800
--
.../slider/api/SliderClusterProtocol.java |6 +-
.../org/apache/slider/api/proto/Messages.java | 1757 +-
.../slider/api/proto/SliderClusterAPI.java | 80 +-
.../client/ipc/SliderClusterOperations.java | 16 +-
.../rpc/SliderClusterProtocolPBImpl.java| 11 +
.../rpc/SliderClusterProtocolProxy.java | 10 +
.../server/appmaster/rpc/SliderIPCService.java |6 +-
.../src/main/proto/SliderClusterMessages.proto | 12 +
.../src/main/proto/SliderClusterProtocol.proto |5 +-
9 files changed, 1893 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b18daa35/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
b/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
index aa3dd3a..b4e91fc 100644
--- a/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
+++ b/slider-core/src/main/java/org/apache/slider/api/SliderClusterProtocol.java
@@ -160,6 +160,8 @@ public interface SliderClusterProtocol extends
VersionedProtocol {
Messages.WrappedJsonProto
getModelResolvedResources(Messages.EmptyPayloadProto request) throws
IOException;
Messages.WrappedJsonProto getLiveResources(Messages.EmptyPayloadProto
request) throws IOException;
-
-
+
+ Messages.SignCertificateResponseProto signCertificate(
+ Messages.SignCertificateRequestProto request) throws IOException;
+
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b18daa35/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
b/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
index 65f243d..3e1cad3 100644
--- a/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
+++ b/slider-core/src/main/java/org/apache/slider/api/proto/Messages.java
@@ -25129,6 +25129,1732 @@ public final class Messages {
//
@@protoc_insertion_point(class_scope:org.apache.slider.api.WrappedJsonProto)
}
+ public interface SignCertificateRequestProtoOrBuilder
+ extends com.google.protobuf.MessageOrBuilder {
+
+// required string hostname = 1;
+/**
+ * coderequired string hostname = 1;/code
+ */
+boolean hasHostname();
+/**
+ * coderequired string hostname = 1;/code
+ */
+java.lang.String getHostname();
+/**
+ * coderequired string hostname = 1;/code
+ */
+com.google.protobuf.ByteString
+getHostnameBytes();
+
+// required string certRequest = 2;
+/**
+ * coderequired string certRequest = 2;/code
+ */
+boolean hasCertRequest();
+/**
+ * coderequired string certRequest = 2;/code
+ */
+java.lang.String getCertRequest();
+/**
+ * coderequired string certRequest = 2;/code
+ */
+com.google.protobuf.ByteString
+getCertRequestBytes();
+
+// required string passPhrase = 3;
+/**
+ * coderequired string passPhrase = 3;/code
+ */
+boolean hasPassPhrase();
+/**
+ * coderequired string passPhrase = 3;/code
+ */
+java.lang.String getPassPhrase();
+/**
+ * coderequired string passPhrase = 3;/code
+ */
+com.google.protobuf.ByteString
+getPassPhraseBytes();
+ }
+ /**
+ * Protobuf type {@code org.apache.slider.api.SignCertificateRequestProto}
+ */
+ public static final class SignCertificateRequestProto extends
+ com.google.protobuf.GeneratedMessage
+ implements SignCertificateRequestProtoOrBuilder {
+// Use SignCertificateRequestProto.newBuilder() to construct.
+private
SignCertificateRequestProto(com.google.protobuf.GeneratedMessage.Builder?
builder) {
+ super(builder);
+ this.unknownFields = builder.getUnknownFields();
+}
+private SignCertificateRequestProto(boolean noInit) { this.unknownFields =
[02/19] incubator-slider git commit: SLIDER-769 Modify Slider app packages for seamless integration with AMS provider (Ambari Metrics Service)
SLIDER-769 Modify Slider app packages for seamless integration with AMS
provider (Ambari Metrics Service)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/24622cbe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/24622cbe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/24622cbe
Branch: refs/heads/feature/package_simplification_II
Commit: 24622cbe5f83c59f3539b50481239f6a88871282
Parents: 54bcc86
Author: Gour Saha gourks...@apache.org
Authored: Tue Mar 3 08:55:01 2015 -0800
Committer: Gour Saha gourks...@apache.org
Committed: Tue Mar 3 08:55:01 2015 -0800
--
app-packages/storm/package/scripts/params.py| 1 +
app-packages/storm/package/templates/config.yaml.j2 | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/24622cbe/app-packages/storm/package/scripts/params.py
--
diff --git a/app-packages/storm/package/scripts/params.py
b/app-packages/storm/package/scripts/params.py
index 8354617..379b229 100644
--- a/app-packages/storm/package/scripts/params.py
+++ b/app-packages/storm/package/scripts/params.py
@@ -43,6 +43,7 @@ storm_env_sh_template =
config['configurations']['storm-env']['content']
metric_collector_host =
default('/configurations/global/metric_collector_host', '')
metric_collector_port =
default('/configurations/global/metric_collector_port', '')
metric_collector_lib = default('/configurations/global/metric_collector_lib',
'')
+metric_collector_app_id = format({app_name})
has_metric_collector = 1
if not metric_collector_lib:
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/24622cbe/app-packages/storm/package/templates/config.yaml.j2
--
diff --git a/app-packages/storm/package/templates/config.yaml.j2
b/app-packages/storm/package/templates/config.yaml.j2
index d9c47a9..5294917 100644
--- a/app-packages/storm/package/templates/config.yaml.j2
+++ b/app-packages/storm/package/templates/config.yaml.j2
@@ -27,9 +27,9 @@ enableMetricsSink: True
metrics_collector:
reportInterval: 60
-
host: {{metric_collector_host}}
port: {{metric_collector_port}}
+ appId: {{metric_collector_app_id}}
{% else %}
enableGanglia: True
Git Push Summary
Repository: incubator-slider Updated Branches: refs/heads/feature/package_simplification_II [created] de41cb5bd
[1/2] incubator-slider git commit: Cleaning up metainfo
Repository: incubator-slider
Updated Branches:
refs/heads/feature/packaging_improvements f3cab4445 - 5b1826e1b
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5b1826e1/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Export.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Export.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Export.java
deleted file mode 100644
index 7331144..000
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Export.java
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.slider.providers.agent.application.metadata.json;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * Represents package description.
- */
-public class Export {
- protected static final Logger
- log = LoggerFactory.getLogger(Export.class);
-
-
- private String name;
- private String value;
-
- /**
- * Creator.
- */
- public Export() {
- }
-
-
- public void setName(String name) {
-this.name = name;
- }
-
- public String getName() {
-return name;
- }
-
- public void setValue(String value) {
-this.value = value;
- }
-
- public String getValue() {
-return value;
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5b1826e1/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ExportGroup.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ExportGroup.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ExportGroup.java
deleted file mode 100644
index e95497d..000
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ExportGroup.java
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.slider.providers.agent.application.metadata.json;
-
-import org.codehaus.jackson.annotate.JsonProperty;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.util.ArrayList;
-import java.util.List;
-
-/**
- * Represents package description.
- */
-public class ExportGroup {
- protected static final Logger
- log = LoggerFactory.getLogger(ExportGroup.class);
-
-
- private String name;
- private ListExport exports = new ArrayListExport();
-
- /**
- * Creator.
- */
- public ExportGroup() {
- }
-
- @JsonProperty(exports)
- public ListExport getExports() {
-return this.exports;
- }
-
- public void setName(String name) {
-this.name = name;
- }
-
- public String getName() {
-return name;
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5b1826e1/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
deleted
[2/2] incubator-slider git commit: Cleaning up metainfo
Cleaning up metainfo Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/5b1826e1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/5b1826e1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/5b1826e1 Branch: refs/heads/feature/packaging_improvements Commit: 5b1826e1b030383ea5e552ef3268334cc654ecff Parents: f3cab44 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Mar 1 00:34:35 2015 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Mar 1 00:34:35 2015 -0800 -- .../providers/agent/AgentClientProvider.java| 40 +++-- .../providers/agent/AgentProviderService.java | 110 +--- .../slider/providers/agent/AgentUtils.java | 22 +-- .../providers/agent/ComponentCommandOrder.java | 2 +- .../agent/application/metadata/Application.java | 94 -- .../application/metadata/CommandOrder.java | 9 +- .../application/metadata/CommandScript.java | 11 +- .../agent/application/metadata/Component.java | 97 -- .../application/metadata/ComponentCommand.java | 72 .../agent/application/metadata/ConfigFile.java | 8 +- .../application/metadata/DockerContainer.java | 91 ++ .../metadata/DockerContainerMount.java | 59 ++ .../metadata/DockerContainerPort.java | 59 ++ .../agent/application/metadata/Export.java | 9 +- .../agent/application/metadata/ExportGroup.java | 11 +- .../agent/application/metadata/Metainfo.java| 32 ++-- .../application/metadata/MetainfoParser.java| 62 ++- .../agent/application/metadata/OSPackage.java | 9 +- .../agent/application/metadata/OSSpecific.java | 11 +- .../agent/application/metadata/Package.java | 60 +++ .../agent/application/metadata/Validate.java| 27 +++ .../application/metadata/json/Application.java | 123 - .../application/metadata/json/CommandOrder.java | 57 -- .../metadata/json/CommandScript.java| 66 --- .../application/metadata/json/Component.java| 144 --- .../metadata/json/ComponentCommand.java | 79 .../application/metadata/json/ConfigFile.java | 53 -- .../application/metadata/json/Container.java| 79 .../metadata/json/ContainerMount.java | 53 -- .../metadata/json/ContainerPort.java| 53 -- .../agent/application/metadata/json/Export.java | 57 -- .../application/metadata/json/ExportGroup.java | 57 -- .../application/metadata/json/MetaInfo.java | 178 --- .../metadata/json/MetaInfoParser.java | 85 - .../application/metadata/json/Package.java | 54 -- .../appmaster/web/rest/agent/Container.java | 82 - .../web/rest/agent/ContainerMount.java | 58 -- .../appmaster/web/rest/agent/ContainerPort.java | 57 -- .../web/rest/agent/ExecutionCommand.java| 25 +-- .../common/tools/TestWindowsSupport.groovy | 2 +- .../agent/TestAgentClientProvider2.java | 8 +- .../agent/TestAgentProviderService.java | 120 + .../agent/TestComponentCommandOrder.java| 2 +- .../metadata/TestMetainfoParser.java| 104 ++- .../metadata/json/TestMetaInfoParser.java | 134 -- 45 files changed, 1003 insertions(+), 1622 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5b1826e1/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java -- diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java index 42ac47d..9978b0a 100644 --- a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java +++ b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java @@ -40,10 +40,10 @@ import org.apache.slider.core.launch.AbstractLauncher; import org.apache.slider.providers.AbstractClientProvider; import org.apache.slider.providers.ProviderRole; import org.apache.slider.providers.ProviderUtils; -import org.apache.slider.providers.agent.application.metadata.json.Application; -import org.apache.slider.providers.agent.application.metadata.json.Component; -import org.apache.slider.providers.agent.application.metadata.json.MetaInfo; -import org.apache.slider.providers.agent.application.metadata.json.MetaInfoParser; +import org.apache.slider.providers.agent.application.metadata.Application; +import org.apache.slider.providers.agent.application.metadata.Component; +import
incubator-slider git commit: Adding support for Docker container
Repository: incubator-slider
Updated Branches:
refs/heads/feature/packaging_improvements c7b08d869 - ec2f013eb
Adding support for Docker container
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/ec2f013e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/ec2f013e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/ec2f013e
Branch: refs/heads/feature/packaging_improvements
Commit: ec2f013eb1b6dddc31902093644c06cf49a4891e
Parents: c7b08d8
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Mar 1 22:39:02 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Mar 1 22:39:02 2015 -0800
--
.../src/main/python/agent/ActionQueue.py| 24 +-
.../src/main/python/agent/DockerManager.py | 178
slider-agent/src/main/python/agent/main.py | 6 +
slider-core/pom.xml | 6 +-
.../providers/agent/AgentProviderService.java | 277 ++-
.../application/metadata/DockerContainer.java | 37 ++-
.../metadata/DockerContainerInputFile.java | 32 +++
.../web/rest/agent/ExecutionCommand.java| 12 +
8 files changed, 559 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/ec2f013e/slider-agent/src/main/python/agent/ActionQueue.py
--
diff --git a/slider-agent/src/main/python/agent/ActionQueue.py
b/slider-agent/src/main/python/agent/ActionQueue.py
index 4cb5de7..e82339e 100644
--- a/slider-agent/src/main/python/agent/ActionQueue.py
+++ b/slider-agent/src/main/python/agent/ActionQueue.py
@@ -31,7 +31,9 @@ from AgentToggleLogger import AgentToggleLogger
from CommandStatusDict import CommandStatusDict
from CustomServiceOrchestrator import CustomServiceOrchestrator
import Constants
-
+import subprocess
+import getpass
+from DockerManager import DockerManager
logger = logging.getLogger()
installScriptHash = -1
@@ -51,6 +53,8 @@ class ActionQueue(threading.Thread):
STORE_APPLIED_CONFIG = 'record_config'
AUTO_RESTART = 'auto_restart'
+
+ docker_mode = False
def __init__(self, config, controller, agentToggleLogger):
super(ActionQueue, self).__init__()
@@ -66,7 +70,8 @@ class ActionQueue(threading.Thread):
self.customServiceOrchestrator = CustomServiceOrchestrator(config,
controller,
self.queueOutAgentToggleLogger)
-
+self.dockerManager = DockerManager(self.tmpdir, config.getWorkRootPath(),
self.customServiceOrchestrator)
+
def stop(self):
self._stop.set()
@@ -157,7 +162,14 @@ class ActionQueue(threading.Thread):
logger.info(Component has indicated auto-restart. Saving details from
START command.)
# running command
-commandresult = self.customServiceOrchestrator.runCommand(command,
+commandresult = None
+logger.info(command fromhost: + str(command))
+
+if 'configurations' in command and 'docker' in command['configurations']:
+self.docker_mode = True
+commandresult = self.dockerManager.execute_command(command,
store_config or store_command)
+else:
+commandresult = self.customServiceOrchestrator.runCommand(command,
in_progress_status[
'tmpout'],
in_progress_status[
@@ -214,7 +226,11 @@ class ActionQueue(threading.Thread):
service = command['serviceName']
component = command['componentName']
reportResult = CommandStatusDict.shouldReportResult(command)
- component_status =
self.customServiceOrchestrator.requestComponentStatus(command)
+ component_status = None
+ if self.docker_mode:
+component_status = self.dockerManager.query_status(command)
+ else:
+component_status =
self.customServiceOrchestrator.requestComponentStatus(command)
result = {componentName: component,
msg: ,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/ec2f013e/slider-agent/src/main/python/agent/DockerManager.py
--
diff --git a/slider-agent/src/main/python/agent/DockerManager.py
b/slider-agent/src/main/python/agent/DockerManager.py
new file mode 100644
index 000..23c8c49
--- /dev/null
+++ b/slider-agent/src/main/python/agent/DockerManager.py
@@ -0,0 +1,178 @@
+import logging
+import os
+import subprocess
+from AgentConfig import AgentConfig
+import Constants
+
+logger =
incubator-slider git commit: Cleaning up metainfo-II
Repository: incubator-slider
Updated Branches:
refs/heads/feature/packaging_improvements 5b1826e1b - c7b08d869
Cleaning up metainfo-II
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/c7b08d86
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/c7b08d86
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/c7b08d86
Branch: refs/heads/feature/packaging_improvements
Commit: c7b08d86904c6448277f2fa58b09deca7a10856a
Parents: 5b1826e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Mar 1 10:48:53 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Mar 1 10:48:53 2015 -0800
--
.../apache/slider/providers/agent/TestAgentClientProvider2.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/c7b08d86/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentClientProvider2.java
--
diff --git
a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentClientProvider2.java
b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentClientProvider2.java
index 92efafd..cdcb878 100644
---
a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentClientProvider2.java
+++
b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentClientProvider2.java
@@ -249,7 +249,7 @@ public class TestAgentClientProvider2 {
client.actionClient(args);
}catch(SliderException e) {
log.info(e.getMessage());
- Assert.assertTrue(e.getMessage().contains(Not a valid app package.
Could not read metainfo.xml));
+ Assert.assertTrue(e.getMessage().contains(Not a valid app package.
Could not read metainfo));
}
}
}
incubator-slider git commit: Package simplification poc-III
Repository: incubator-slider
Updated Branches:
refs/heads/feature/packaging_improvements 82dd48373 - f3cab4445
Package simplification poc-III
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/f3cab444
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/f3cab444
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/f3cab444
Branch: refs/heads/feature/packaging_improvements
Commit: f3cab444587dfaede819c4635ddccc9778410b7d
Parents: 82dd483
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Feb 25 18:29:54 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Feb 25 18:29:54 2015 -0800
--
.../apache/slider/providers/agent/AgentProviderService.java| 2 --
.../providers/agent/application/metadata/json/MetaInfo.java| 1 +
.../slider/providers/agent/TestAgentProviderService.java | 6 +++---
3 files changed, 4 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3cab444/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index d4bb0f4..b98795a 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -1644,7 +1644,6 @@ public class AgentProviderService extends
AbstractProviderService implements
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
-cmd.addContainerDetails(componentName, getMetaInfo());
response.addExecutionCommand(cmd);
}
@@ -1827,7 +1826,6 @@ public class AgentProviderService extends
AbstractProviderService implements
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
-cmd.addContainerDetails(componentName, getMetaInfo());
response.addExecutionCommand(cmd);
// With start command, the corresponding command for graceful stop needs to
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3cab444/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
index 8938ee0..ff81fb9 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/MetaInfo.java
@@ -155,6 +155,7 @@ public class MetaInfo {
commandScript.setTimeout(xmlComp.getCommandScript().getTimeout());
comp.setCommandScript(commandScript);
}
+ app.getComponents().add(comp);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3cab444/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
--
diff --git
a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
index 0324af4..7746867 100644
---
a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
+++
b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
@@ -993,10 +993,10 @@ public class TestAgentProviderService {
int found = 0;
for (Component component : components) {
if (component.getName().equals(HBASE_MASTER)) {
-Assert.assertEquals(component.getAutoStartOnFailure(), true);
Assert.assertEquals(component.getAutoStartOnFailure(), Boolean.TRUE);
-Assert.assertEquals(component.getMinInstanceCount(), 1);
-Assert.assertEquals(component.getMaxInstanceCount(), 2);
+Assert.assertEquals(component.getAutoStartOnFailure(), Boolean.TRUE);
+Assert.assertEquals(component.getMinInstanceCount(), 1);
+Assert.assertEquals(component.getMaxInstanceCount(), 2);
Assert.assertEquals(component.getCommandScript().getScript(),
scripts/hbase_master.py);
Assert.assertEquals(component.getCategory(), MASTER);
incubator-slider git commit: Package simplification poc-II
Repository: incubator-slider
Updated Branches:
refs/heads/feature/packaging_improvements a31d20eed - 82dd48373
Package simplification poc-II
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/82dd4837
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/82dd4837
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/82dd4837
Branch: refs/heads/feature/packaging_improvements
Commit: 82dd48373134a17fba55a48f3f3928dfce3f8680
Parents: a31d20e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Feb 25 13:25:15 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Feb 25 13:25:15 2015 -0800
--
.../providers/agent/AgentProviderService.java | 5 ++
.../application/metadata/json/Component.java| 15
.../application/metadata/json/Container.java| 79 +++
.../metadata/json/ContainerMount.java | 53 +
.../metadata/json/ContainerPort.java| 53 +
.../appmaster/web/rest/agent/Container.java | 82
.../web/rest/agent/ContainerMount.java | 58 ++
.../appmaster/web/rest/agent/ContainerPort.java | 57 ++
.../web/rest/agent/ExecutionCommand.java| 56 -
.../metadata/json/TestMetaInfoParser.java | 19 +
10 files changed, 475 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/82dd4837/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index 4104cdd..d4bb0f4 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -1643,6 +1643,8 @@ public class AgentProviderService extends
AbstractProviderService implements
cmd.setCommandParams(setCommandParameters(scriptPath, timeout, false));
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
+
+cmd.addContainerDetails(componentName, getMetaInfo());
response.addExecutionCommand(cmd);
}
@@ -1682,6 +1684,7 @@ public class AgentProviderService extends
AbstractProviderService implements
configurations.get(global).put(exec_cmd, effectiveCommand.getExec());
cmd.setHostname(getClusterInfoPropertyValue(StatusKeys.INFO_AM_HOSTNAME));
+cmd.addContainerDetails(componentName, getMetaInfo());
response.addExecutionCommand(cmd);
}
@@ -1824,6 +1827,7 @@ public class AgentProviderService extends
AbstractProviderService implements
MapString, MapString, String configurations =
buildCommandConfigurations(appConf, containerId, componentName);
cmd.setConfigurations(configurations);
+cmd.addContainerDetails(componentName, getMetaInfo());
response.addExecutionCommand(cmd);
// With start command, the corresponding command for graceful stop needs to
@@ -1891,6 +1895,7 @@ public class AgentProviderService extends
AbstractProviderService implements
cmd.setConfigurations(configurations);
configurations.get(global).put(exec_cmd, startCommand.getExec());
+cmd.addContainerDetails(componentName, getMetaInfo());
response.addExecutionCommand(cmd);
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/82dd4837/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Component.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Component.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Component.java
index 8697e48..84147a3 100644
---
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Component.java
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/Component.java
@@ -40,12 +40,14 @@ public class Component {
private String compExports;
private String category = MASTER;
private String publishConfig;
+ private String type;
private int minInstanceCount = 1;
private int maxInstanceCount = Integer.MAX_VALUE;
private Boolean autoStartOnFailure = false;
private String appExports;
private CommandScript commandScript;
private ListComponentCommand commands = new ArrayListComponentCommand();
+ private ListContainer containers = new ArrayListContainer();
public
[2/2] incubator-slider git commit: Package simplification poc-I
Package simplification poc-I
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a31d20ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a31d20ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a31d20ee
Branch: refs/heads/feature/packaging_improvements
Commit: a31d20eedd323209d7d6116b67aa7b2eecf3b39a
Parents: 8d0e337
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Feb 24 20:17:00 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Feb 24 20:17:00 2015 -0800
--
pom.xml | 2 +-
.../python/agent/CustomServiceOrchestrator.py | 87 +-
.../src/main/python/scripts/basic_installer.py | 55
slider-agent/src/main/python/scripts/params.py | 29 ++
.../org/apache/slider/api/InternalKeys.java | 8 +
.../org/apache/slider/client/SliderClient.java | 147 +-
.../org/apache/slider/common/SliderKeys.java| 12 +-
.../AbstractClusterBuildingActionArgs.java | 31 +-
.../slider/common/params/AddonArgsDelegate.java | 54
.../apache/slider/common/params/Arguments.java | 3 +
.../slider/common/tools/CoreFileSystem.java | 27 +-
.../apache/slider/common/tools/SliderUtils.java | 56 +++-
.../slider/core/build/InstanceBuilder.java | 6 +
.../slider/core/persist/InstancePaths.java | 4 +
.../providers/agent/AgentClientProvider.java| 29 +-
.../slider/providers/agent/AgentKeys.java | 2 +
.../providers/agent/AgentProviderService.java | 284 ---
.../slider/providers/agent/AgentUtils.java | 38 ++-
.../providers/agent/ComponentCommandOrder.java | 6 +-
.../agent/application/metadata/Metainfo.java| 12 +
.../application/metadata/json/Application.java | 123
.../application/metadata/json/CommandOrder.java | 57
.../metadata/json/CommandScript.java| 66 +
.../application/metadata/json/Component.java| 129 +
.../metadata/json/ComponentCommand.java | 79 ++
.../application/metadata/json/ConfigFile.java | 53
.../agent/application/metadata/json/Export.java | 57
.../application/metadata/json/ExportGroup.java | 57
.../application/metadata/json/MetaInfo.java | 177
.../metadata/json/MetaInfoParser.java | 85 ++
.../application/metadata/json/Package.java | 54
.../common/tools/TestWindowsSupport.groovy | 2 +-
.../agent/TestAgentClientProvider2.java | 16 +-
.../agent/TestAgentProviderService.java | 131 +++--
.../agent/TestComponentCommandOrder.java| 2 +-
.../metadata/json/TestMetaInfoParser.java | 115
36 files changed, 1830 insertions(+), 265 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a31d20ee/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 75e60bd..6c5f2d5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -112,7 +112,7 @@
!--
Java versions
--
-project.java.src.version1.6/project.java.src.version
+project.java.src.version1.7/project.java.src.version
enforced.java.version${project.java.src.version}/enforced.java.version
groovy.version2.2.2/groovy.version
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a31d20ee/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
--
diff --git a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
index 119c926..51ce975 100644
--- a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
+++ b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
@@ -76,23 +76,36 @@ class CustomServiceOrchestrator():
override_output_files=True, store_command=False):
allocated_ports = {}
try:
+ py_file_list = []
+ json_path = None
+
script_type = command['commandParams']['script_type']
- script = command['commandParams']['script']
- timeout = int(command['commandParams']['command_timeout'])
task_id = command['taskId']
command_name = command['roleCommand']
- script_path = self.resolve_script_path(self.base_dir, script,
script_type)
- script_tuple = (script_path, self.base_dir)
-
tmpstrucoutfile = os.path.realpath(posixpath.join(self.tmp_dir,
structured-out-{0}.json.format(task_id)))
- if script_type.upper() != self.SCRIPT_TYPE_PYTHON:
- # We don't support anything else yet
+ if script_type.upper() == self.SCRIPT_TYPE_PYTHON:
+
[1/2] incubator-slider git commit: Package simplification poc-I
Repository: incubator-slider
Updated Branches:
refs/heads/feature/packaging_improvements 8d0e337a0 - a31d20eed
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a31d20ee/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ComponentCommand.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ComponentCommand.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ComponentCommand.java
new file mode 100644
index 000..aa7f3c3
--- /dev/null
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ComponentCommand.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.slider.providers.agent.application.metadata.json;
+
+import org.codehaus.jackson.JsonParseException;
+import org.codehaus.jackson.annotate.JsonIgnore;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.annotate.JsonProperty;
+import org.codehaus.jackson.map.JsonMappingException;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.map.SerializationConfig;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Represents the metadata associated with the application.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+public class ComponentCommand {
+ protected static final Logger
+ log = LoggerFactory.getLogger(ComponentCommand.class);
+
+
+ private String exec;
+ private String name = START;
+ private String type = SHELL;
+
+ /**
+ * Creator.
+ */
+ public ComponentCommand() {
+ }
+
+ public void setName(String name) {
+this.name = name;
+ }
+
+ public String getName() {
+return name;
+ }
+
+ public void setExec(String exec) {
+this.exec = exec;
+ }
+
+ public String getExec() {
+return exec;
+ }
+
+ public void setType(String type) {
+this.type = type;
+ }
+
+ public String getType() {
+return type;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a31d20ee/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ConfigFile.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ConfigFile.java
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ConfigFile.java
new file mode 100644
index 000..9a83f35
--- /dev/null
+++
b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/json/ConfigFile.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.slider.providers.agent.application.metadata.json;
+
+/**
+ *
+ */
+public class ConfigFile {
+ String type;
+ String fileName;
+ String dictionaryName;
+
+ public ConfigFile() {
+ }
+
+ public String getType() {
+return type;
+ }
+
+ public void setType(String type) {
+this.type = type;
+ }
+
+ public String getFileName() {
+return fileName;
+ }
+
+ public void setFileName(String fileName) {
incubator-slider git commit: SLIDER-763. Add install client command to slider
Repository: incubator-slider
Updated Branches:
refs/heads/develop e23893cdd - 522983bd9
SLIDER-763. Add install client command to slider
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/522983bd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/522983bd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/522983bd
Branch: refs/heads/develop
Commit: 522983bd9a760110697ccbd93dfd5c29872a1656
Parents: e23893c
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Feb 19 13:38:12 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Feb 19 13:40:51 2015 -0800
--
.../slider-pkg/clientInstallConfig-default.json | 8 +
.../command-logger/slider-pkg/metainfo.xml | 10 +
.../slider-pkg/package/scripts/cl_client.py | 48 +++
.../slider-pkg/package/scripts/client_params.py | 29 ++
.../storm/clientInstallConfig-default.json | 6 +
app-packages/storm/metainfo.xml | 10 +
app-packages/storm/package/files/storm-slider | 63
.../storm/package/files/storm-slider.py | 370 +++
.../storm/package/scripts/client_params.py | 28 ++
.../storm/package/scripts/storm_client.py | 56 +++
.../package/templates/storm-slider-env.sh.j2| 38 ++
app-packages/storm/src/assembly/storm.xml | 7 +
.../org/apache/slider/client/SliderClient.java | 66 +++-
.../apache/slider/client/SliderClientAPI.java | 14 +-
.../slider/common/params/ActionClientArgs.java | 65
.../slider/common/params/ActionPackageArgs.java | 4 +-
.../apache/slider/common/params/Arguments.java | 3 +-
.../apache/slider/common/params/ClientArgs.java | 19 +-
.../slider/common/params/SliderActions.java | 3 +
.../providers/AbstractClientProvider.java | 20 +
.../providers/agent/AgentClientProvider.java| 304 ++-
.../providers/agent/AgentProviderService.java | 7 +-
.../test_command_log/client-config.json | 7 +
.../agent/actions/TestActionPackage.groovy | 20 +-
.../agent/TestAgentClientProvider2.java | 173 -
.../lifecycle/AgentClientInstallIT.groovy | 79
.../clusters/remote/slider/slider-client.xml| 10 +-
27 files changed, 1434 insertions(+), 33 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/522983bd/app-packages/command-logger/slider-pkg/clientInstallConfig-default.json
--
diff --git
a/app-packages/command-logger/slider-pkg/clientInstallConfig-default.json
b/app-packages/command-logger/slider-pkg/clientInstallConfig-default.json
new file mode 100644
index 000..f0b84a5
--- /dev/null
+++ b/app-packages/command-logger/slider-pkg/clientInstallConfig-default.json
@@ -0,0 +1,8 @@
+{
+ schema:http://example.org/specification/v2.0.0;,
+ global:{
+client_root:{app_install_dir},
+application_id: application_id_1
+ }
+}
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/522983bd/app-packages/command-logger/slider-pkg/metainfo.xml
--
diff --git a/app-packages/command-logger/slider-pkg/metainfo.xml
b/app-packages/command-logger/slider-pkg/metainfo.xml
index 5de2c37..eb02f6a 100644
--- a/app-packages/command-logger/slider-pkg/metainfo.xml
+++ b/app-packages/command-logger/slider-pkg/metainfo.xml
@@ -36,6 +36,16 @@
timeout600/timeout
/commandScript
/component
+
+ component
+nameCOMMAND_LOGGER_CLIENT/name
+categoryCLIENT/category
+commandScript
+ scriptscripts/cl_client.py/script
+ scriptTypePYTHON/scriptType
+ timeout600/timeout
+/commandScript
+ /component
/components
osSpecifics
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/522983bd/app-packages/command-logger/slider-pkg/package/scripts/cl_client.py
--
diff --git
a/app-packages/command-logger/slider-pkg/package/scripts/cl_client.py
b/app-packages/command-logger/slider-pkg/package/scripts/cl_client.py
new file mode 100644
index 000..73a2675
--- /dev/null
+++ b/app-packages/command-logger/slider-pkg/package/scripts/cl_client.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+License); you may not use this file except in compliance
+with the License. You may obtain a copy of the
incubator-slider git commit: SLIDER-763. Add install client command to slider (rename default config file)
Repository: incubator-slider Updated Branches: refs/heads/develop df9d5adb9 - 5f5d8dfb5 SLIDER-763. Add install client command to slider (rename default config file) Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/5f5d8dfb Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/5f5d8dfb Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/5f5d8dfb Branch: refs/heads/develop Commit: 5f5d8dfb531b3b96469a5978c421157f8adb2d99 Parents: df9d5ad Author: Sumit Mohanty smoha...@hortonworks.com Authored: Thu Feb 19 23:39:46 2015 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Thu Feb 19 23:39:46 2015 -0800 -- app-packages/storm/src/assembly/storm.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5f5d8dfb/app-packages/storm/src/assembly/storm.xml -- diff --git a/app-packages/storm/src/assembly/storm.xml b/app-packages/storm/src/assembly/storm.xml index 6a6e750..4e13650 100644 --- a/app-packages/storm/src/assembly/storm.xml +++ b/app-packages/storm/src/assembly/storm.xml @@ -42,7 +42,7 @@ fileMode0755/fileMode /file file - sourceclientInstall-default.json/source + sourceclientInstallConfig-default.json/source outputDirectory//outputDirectory filteredtrue/filtered fileMode0755/fileMode @@ -71,7 +71,7 @@ excludetarget/**/exclude excludeappConfig-default.json/exclude excludeappConfig-secured-default.json/exclude -excludeclientInstall-default.json/exclude +excludeclientInstallConfig-default.json/exclude excludemetainfo.xml/exclude /excludes fileMode0755/fileMode
incubator-slider git commit: SLIDER-776. additional_cp is not necessary for memcached (Yang Hao via smohanty)
Repository: incubator-slider
Updated Branches:
refs/heads/develop ba19e010f - 790ef0e4a
SLIDER-776. additional_cp is not necessary for memcached (Yang Hao via smohanty)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/790ef0e4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/790ef0e4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/790ef0e4
Branch: refs/heads/develop
Commit: 790ef0e4a16ffe2feae33b8dbc6dc8487f3c37fd
Parents: ba19e01
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Feb 4 15:43:31 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Feb 4 15:43:31 2015 -0800
--
app-packages/memcached/package/scripts/params.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/790ef0e4/app-packages/memcached/package/scripts/params.py
--
diff --git a/app-packages/memcached/package/scripts/params.py
b/app-packages/memcached/package/scripts/params.py
index 056a3b9..970638a 100644
--- a/app-packages/memcached/package/scripts/params.py
+++ b/app-packages/memcached/package/scripts/params.py
@@ -27,7 +27,7 @@ app_root = config['configurations']['global']['app_root']
java64_home = config['hostLevelParams']['java_home']
pid_file = config['configurations']['global']['pid_file']
-additional_cp = config['configurations']['global']['additional_cp']
+additional_cp = default('/configurations/global/additional_cp', '')
xmx_val = config['configurations']['global']['xmx_val']
xms_val = config['configurations']['global']['xms_val']
memory_val = config['configurations']['global']['memory_val']
incubator-slider git commit: SLIDER-743. Include node failure history when choosing placement hints
Repository: incubator-slider
Updated Branches:
refs/heads/develop 922439e58 - 39e04e36e
SLIDER-743. Include node failure history when choosing placement hints
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/39e04e36
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/39e04e36
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/39e04e36
Branch: refs/heads/develop
Commit: 39e04e36ed72e8a40434e82db0d7f0f210f26f1d
Parents: 922439e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Fri Jan 30 16:54:26 2015 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Fri Jan 30 16:54:26 2015 -0800
--
.../org/apache/slider/api/ResourceKeys.java | 13 -
.../apache/slider/providers/ProviderRole.java | 11 ++--
.../slideram/SliderAMClientProvider.java| 3 +-
.../server/appmaster/SliderAppMaster.java | 1 -
.../slider/server/appmaster/state/AppState.java | 26 --
.../appmaster/state/OutstandingRequest.java | 20 --
.../server/appmaster/state/RoleHistory.java | 4 +--
.../server/appmaster/state/RoleStatus.java | 9 ++
.../TestMockAppStateDynamicRoles.groovy | 9 ++
.../TestRoleHistoryRequestTracking.groovy | 29 +++-
10 files changed, 110 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/39e04e36/slider-core/src/main/java/org/apache/slider/api/ResourceKeys.java
--
diff --git a/slider-core/src/main/java/org/apache/slider/api/ResourceKeys.java
b/slider-core/src/main/java/org/apache/slider/api/ResourceKeys.java
index 52633f4..50ca82f 100644
--- a/slider-core/src/main/java/org/apache/slider/api/ResourceKeys.java
+++ b/slider-core/src/main/java/org/apache/slider/api/ResourceKeys.java
@@ -100,7 +100,11 @@ public interface ResourceKeys {
*/
String COMPONENT_PLACEMENT_POLICY = yarn.component.placement.policy;
-
+ /**
+ * Maximum number of node failures that can be tolerated by a component on a
specific node
+ */
+ String NODE_FAILURE_THRESHOLD =
+ yarn.node.failure.threshold;
/**
* maximum number of failed containers (in a single role)
@@ -130,6 +134,13 @@ public interface ResourceKeys {
int DEFAULT_CONTAINER_FAILURE_THRESHOLD = 5;
/**
+ * Default node failure threshold for a component instance: {@value}
+ * Should to be lower than default component failure threshold to allow
+ * the component to start elsewhere
+ */
+ int DEFAULT_NODE_FAILURE_THRESHOLD = 3;
+
+ /**
* Log aggregation include, exclude patterns
*/
String YARN_LOG_INCLUDE_PATTERNS = yarn.log.include.patterns;
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/39e04e36/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
b/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
index 5b85f7b..17124d2 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
@@ -18,6 +18,8 @@
package org.apache.slider.providers;
+import org.apache.slider.api.ResourceKeys;
+
/**
* Provider role and key for use in app requests.
*
@@ -28,15 +30,17 @@ public final class ProviderRole {
public final String name;
public final int id;
public final int placementPolicy;
+ public final int nodeFailureThreshold;
public ProviderRole(String name, int id) {
-this(name, id, PlacementPolicy.DEFAULT);
+this(name, id, PlacementPolicy.DEFAULT,
ResourceKeys.DEFAULT_NODE_FAILURE_THRESHOLD);
}
- public ProviderRole(String name, int id, int policy) {
+ public ProviderRole(String name, int id, int policy, int
nodeFailureThreshold) {
this.name = name;
this.id = id;
this.placementPolicy = policy;
+this.nodeFailureThreshold = nodeFailureThreshold;
}
@Override
@@ -59,10 +63,11 @@ public final class ProviderRole {
@Override
public String toString() {
-return ProviderRole{ +
+return ProviderRole { +
name=' + name + '\'' +
, id= + id +
, policy= + placementPolicy +
+ , nodeFailureThreshold= + nodeFailureThreshold +
'}';
}
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/39e04e36/slider-core/src/main/java/org/apache/slider/providers/slideram/SliderAMClientProvider.java
--
diff --git
svn commit: r938280 - /websites/production/slider/content/
Author: smohanty Date: Thu Jan 29 20:31:59 2015 New Revision: 938280 Log: Publishing svnmucc operation to slider site by smohanty Added: websites/production/slider/content/ - copied from r938279, websites/staging/slider/trunk/content/
svn commit: r1641948 - /incubator/slider/site/trunk/content/release_notes/release-0.60.0.md
Author: smohanty Date: Wed Nov 26 21:59:20 2014 New Revision: 1641948 URL: http://svn.apache.org/r1641948 Log: Updating rel doc for 0.60 Modified: incubator/slider/site/trunk/content/release_notes/release-0.60.0.md Modified: incubator/slider/site/trunk/content/release_notes/release-0.60.0.md URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/release_notes/release-0.60.0.md?rev=1641948r1=1641947r2=1641948view=diff == --- incubator/slider/site/trunk/content/release_notes/release-0.60.0.md (original) +++ incubator/slider/site/trunk/content/release_notes/release-0.60.0.md Wed Nov 26 21:59:20 2014 @@ -24,11 +24,28 @@ This release is built against Apache Had ## Key changes -TBD +* Developed against Apache Hadoop 2.6 + +* Integration with Apache Hadooop YARN Service registry for dynamic lookup of URL and configurations + +* Application deployment on Kerberized Hadoop cluster + +* Application deployment on Hadoop on Windows + +* Failure handling such as AppMaster restart, container restart, node failure + +* Integration with Apache Hadoop YARN node labels + +* Log aggregation + +* Integration with Apache Ambari + +* Improved diagnostics support + ## Incompatible Changes -TBD +* Apache Hadoop versions below 2.6 are not supported ## Other changes
svn commit: r1640678 - in /incubator/slider/site/trunk: content/docs/slider_specs/ templates/
Author: smohanty Date: Thu Nov 20 05:28:23 2014 New Revision: 1640678 URL: http://svn.apache.org/r1640678 Log: Updating doc for 0.60 Modified: incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md incubator/slider/site/trunk/content/docs/slider_specs/creating_app_definitions.md incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md incubator/slider/site/trunk/templates/nav.html Modified: incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md?rev=1640678r1=1640677r2=1640678view=diff == --- incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md (original) +++ incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md Thu Nov 20 05:28:23 2014 @@ -39,7 +39,9 @@ The fields to describe an application is * *commandScript*: application wide commands may also be defined. The command is executed on a component instance that is a client -* *dependencies*: application can define a list of dependencies. Dependencies can be on the base services such as HDFS, ZOOKEEPER, YARN which are infrastructure services or GANGLIA, NAGIOS, etc. which are monitoring/alert services. The dependencies are parsed by the management infrastructure to provide the necessary configurations to allow the app to access the services. For example, a HDFS folder could be requested by the app to store its data, a ZOOKEEPER node to co-ordinate among components. +* *comment*: describe the application + +* *exportedConfigs*: what configs to export An application contains several component. The fields associated with a component are: @@ -47,19 +49,33 @@ An application contains several componen * **category**: type of the component - MASTER, SLAVE, and CLIENT -* **minInstanceCount**: the minimum number of instances required for this component +* *minInstanceCount*: the minimum number of instances required for this component * *maxInstanceCount*: maximum number of instances allowed for a component -* **commandScript**: the script that implements the commands. +* *appExports*: which application level exports are exported by this component + +* *compExports*: which component level exports are exported by this component + + A component may contain component specific exports. Refer to [Specifying Exports](/docs/slider_specs/specifying_exports.html) for more details. + + * **name**: the name of the export + + * **value**: the template that will be populated by Slider and then exported + + A component contains command script for the component. - * **script**: the script location - relative to the AppPackage root + * **commandScript**: the script that implements the commands. - * **scriptType**: type of the script +* **script**: the script location - relative to the AppPackage root - * **timeout**: default timeout of the script +* **scriptType**: type of the script -* *customCommands*: any additional commands available for the component and their implementation +* **timeout**: default timeout of the script + + Custom commands may be added to a component. + + * *customCommands*: any additional commands available for the component and their implementation An application definition also includes the package used to install the application. Its typically a tarball or some other form of package that does not require root access to install. The details of what happens during install is captured in the command script. @@ -85,98 +101,80 @@ Applications can also advertise a set of * **name**: specifies the name of the export group -Each exportGroup contains one or more exports. +Each exportGroup contains one or more exports. Refer to [Specifying Exports](/docs/slider_specs/specifying_exports.html) for more details. * **name**: the name of the export * **value**: the template that will be populated by Slider and then exported +Sample metainfo.xml - metainfo -schemaVersion2.0/schemaVersion -application - nameHBASE/name - version0.96.0.2.1.1/version - typeYARN-APP/type - minHadoopVersion2.1.0/minHadoopVersion - components -component - nameHBASE_MASTER/name - categoryMASTER/category - minInstanceCount1/minInstanceCount - maxInstanceCount2/maxInstanceCount - commandScript -scriptscripts/hbase_master.py/script -scriptTypePYTHON/scriptType -timeout600/timeout - /commandScript - customCommands -customCommand
svn commit: r929822 - /websites/production/slider/content/
Author: smohanty Date: Thu Nov 20 05:35:22 2014 New Revision: 929822 Log: Publishing svnmucc operation to slider site by smohanty Added: websites/production/slider/content/ - copied from r929821, websites/staging/slider/trunk/content/
svn commit: r1639803 - in /incubator/slider/site/trunk/content/docs: examples.md getting_started.md slider_specs/specifying_exports.md
Author: smohanty
Date: Fri Nov 14 23:21:36 2014
New Revision: 1639803
URL: http://svn.apache.org/r1639803
Log:
Update to exports documentation and some more clarification on Getting Started
doc
Modified:
incubator/slider/site/trunk/content/docs/examples.md
incubator/slider/site/trunk/content/docs/getting_started.md
incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md
Modified: incubator/slider/site/trunk/content/docs/examples.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/examples.md?rev=1639803r1=1639802r2=1639803view=diff
==
--- incubator/slider/site/trunk/content/docs/examples.md (original)
+++ incubator/slider/site/trunk/content/docs/examples.md Fri Nov 14 23:21:36
2014
@@ -17,6 +17,9 @@
# Apache Slider Examples
+`Note: These examples use HBase providers written for Slider.`
+To use the HBase application package refer to [Getting
Started](/docs/getting_started.html).
+
## Setup
Modified: incubator/slider/site/trunk/content/docs/getting_started.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/getting_started.md?rev=1639803r1=1639802r2=1639803view=diff
==
--- incubator/slider/site/trunk/content/docs/getting_started.md (original)
+++ incubator/slider/site/trunk/content/docs/getting_started.md Fri Nov 14
23:21:36 2014
@@ -204,6 +204,9 @@ Create one or more Slider application pa
slider install-package --name *package name* --package
*sample-application-package*
+Package gets deployed on HDFS at `User home dir/.slider/package/name
provided in the command`
+This path is also reflected in appConfig.json through property
`application.def` So the path and the property value must be in sync.
+
For example, you can use the following command for HBase application package.
slider install-package --name HBASE --package path-to-hbase-package.zip
Modified:
incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md?rev=1639803r1=1639802r2=1639803view=diff
==
--- incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md
(original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md
Fri Nov 14 23:21:36 2014
@@ -29,8 +29,10 @@ All exports are specified in the metadat
Slider application packages accept an appConfig.json file for all application
configuration supplied by the user. Any property whose name starts with site
are considered configuration. [Specifications
InstanceConfiguration](application_instance_configuration.html) describes the
naming convention.
+### Export specific configs
By default all configurations are exported (e.g.
http://hos1:44500/ws/v1/slider/publisher/slider/storm-site). They can be
disabled by specifying `exportedConfigsNone/exportedConfigs` under
`application`. If you want to explicitly specify what to publish you can use
comma separated named such as
`exportedConfigsstorm-site,another-site/exportedConfigs`.
+### Which component is responsible for export
By default an arbitrary master is chosen as the master responsible for
exporting the config. *What this means is that when this master is STARTED the
applied config known at that time is exported*. Otherwise, you can specify
which master component type should export configuration by specifying
`publishConfigtrue/publishConfig` under `component`.
## Exporting formatted data at application instance level
@@ -41,7 +43,7 @@ At application level one or more exportG
nameQuickLinks/name
exports
export
-nameapp.monitor/name
+nameorg.apache.slider.monitor/name
valuehttp://${STORM_UI_SERVER_HOST}:${site.storm-site.ui.port}/value
/export
export
@@ -54,21 +56,107 @@ At application level one or more exportG
You also need to specify as to which component type should export which data
item. This is specified via `appExports` tag such as
`appExportsQuickLinks-nimbus.host_port/appExports` for NIMBUS component
type.
+components
+ component
+nameNIMBUS/name
+categoryMASTER/category
+
appExportsQuickLinks-org.apache.slider.monitor,QuickLinks-nimbus.host_port/appExports
+...
+
## Exporting formatted data at component instance level
-Occasionally, component instances may need to export data on a per component
instance basis. This is true if your application is a set of peers - e.g.
memcached daemons, or you may have a need for each slave component instance to
export some data. Currently, only the host and dynamically allocated port
values can be exported
svn commit: r929269 - /websites/production/slider/content/
Author: smohanty Date: Fri Nov 14 23:25:11 2014 New Revision: 929269 Log: Publishing svnmucc operation to slider site by smohanty Added: websites/production/slider/content/ - copied from r929268, websites/staging/slider/trunk/content/
incubator-slider git commit: Revert Put back detection of failed node
Repository: incubator-slider
Updated Branches:
refs/heads/develop 7d7720881 - 31cdeaca9
Revert Put back detection of failed node
This reverts commit d511022bd3d69386dc32c37db5e7a39126201206.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/31cdeaca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/31cdeaca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/31cdeaca
Branch: refs/heads/develop
Commit: 31cdeaca99ae524c09c685a545d2966079c04d9c
Parents: 7d77208
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Nov 13 17:53:04 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 13 17:53:04 2014 -0800
--
.../org/apache/slider/server/appmaster/state/RoleHistory.java | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/31cdeaca/slider-core/src/main/java/org/apache/slider/server/appmaster/state/RoleHistory.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/RoleHistory.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/RoleHistory.java
index f57ca3f..ce2ab0a 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/RoleHistory.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/RoleHistory.java
@@ -750,11 +750,7 @@ public class RoleHistory {
available = false;
} else {
available = nodeEntry.containerCompleted(wasReleased);
- boolean isFailedNode = failedNodes.contains(RoleHistoryUtils
-.hostnameOf(container));
- if (!isFailedNode) {
-maybeQueueNodeForWork(container, nodeEntry, available);
- }
+ maybeQueueNodeForWork(container, nodeEntry, available);
}
touch();
return available;
incubator-slider git commit: SLIDER-647 allocation requests not being satisfied when a cluster goes to labels
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 0cb6eaf76 - c7186e13d
SLIDER-647 allocation requests not being satisfied when a cluster goes to labels
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/c7186e13
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/c7186e13
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/c7186e13
Branch: refs/heads/releases/slider-0.60
Commit: c7186e13ddd2c1e45e4faf84119b5bb2d04fd00e
Parents: 0cb6eaf
Author: Steve Loughran ste...@apache.org
Authored: Thu Nov 13 22:35:32 2014 +
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 13 18:19:30 2014 -0800
--
.../slider/providers/PlacementPolicy.java | 22 ++-
.../slider/server/appmaster/state/AppState.java | 22 ++-
.../appmaster/state/OutstandingRequest.java | 2 +-
.../server/appmaster/state/RoleHistory.java | 6 +-
.../server/appmaster/state/RoleStatus.java | 4 +
.../TestMockAppStateDynamicHistory.groovy | 3 +-
.../TestMockAppStateDynamicRoles.groovy | 172 ++-
.../TestMockAppStateRolePlacement.groovy| 5 +
.../model/mock/BaseMockAppStateTest.groovy | 2 +
.../appmaster/model/mock/MockYarnCluster.groovy | 10 +-
.../appmaster/model/mock/MockYarnEngine.groovy | 6 +
11 files changed, 229 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/c7186e13/slider-core/src/main/java/org/apache/slider/providers/PlacementPolicy.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/providers/PlacementPolicy.java
b/slider-core/src/main/java/org/apache/slider/providers/PlacementPolicy.java
index 444c041..dc6c910 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/PlacementPolicy.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/PlacementPolicy.java
@@ -23,9 +23,29 @@ package org.apache.slider.providers;
*/
public class PlacementPolicy {
+ /**
+ * Default values
+ */
public static final int DEFAULT = 0;
- public static final int EXCLUDE_FROM_FLEXING = 1;
+
+ /**
+ * Strict placement: when asking for an instance for which there is
+ * history, mandate that it is strict
+ */
+ public static final int STRICT = 1;
+
+ /**
+ * No data locality; do not bother trying to ask for any location
+ */
public static final int NO_DATA_LOCALITY = 2;
+ /**
+ * Anti-affinity is mandatory.
+ */
public static final int ANTI_AFFINITY_REQUIRED = 4;
+
+ /**
+ * Exclude from flexing; used internally to mark AMs.
+ */
+ public static final int EXCLUDE_FROM_FLEXING = 16;
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/c7186e13/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
index 0848173..9956db2 100644
---
a/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
+++
b/slider-core/src/main/java/org/apache/slider/server/appmaster/state/AppState.java
@@ -58,6 +58,7 @@ import org.apache.slider.core.exceptions.ErrorStrings;
import org.apache.slider.core.exceptions.NoSuchNodeException;
import org.apache.slider.core.exceptions.SliderInternalStateException;
import org.apache.slider.core.exceptions.TriggerClusterTeardownException;
+import org.apache.slider.providers.PlacementPolicy;
import org.apache.slider.providers.ProviderRole;
import org.apache.slider.server.appmaster.operations.AbstractRMOperation;
import org.apache.slider.server.appmaster.operations.CancelRequestOperation;
@@ -582,13 +583,15 @@ public class AppState {
int priority = SliderUtils.parseAndValidate(value of + name + +
ResourceKeys.COMPONENT_PRIORITY,
priOpt, 0, 1, -1);
-log.info(Role {} assigned priority {}, name, priority);
String placementOpt = component.getOption(
- ResourceKeys.COMPONENT_PLACEMENT_POLICY, 0);
+ ResourceKeys.COMPONENT_PLACEMENT_POLICY,
+Integer.toString(PlacementPolicy.DEFAULT));
int placement = SliderUtils.parseAndValidate(value of + name + +
ResourceKeys.COMPONENT_PLACEMENT_POLICY,
placementOpt, 0, 0, -1);
-return new ProviderRole(name, priority, placement);
+ProviderRole newRole = new ProviderRole(name, priority, placement);
+log.info(New {} , newRole);
+return newRole;
}
@@ -998,17 +1001,17 @@ public class AppState {
/**
- * enum nodes by role ID,
incubator-slider git commit: SLIDER-555. Default ZK node is not getting deleted when you destroy an application
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 c7186e13d - 2c6cc4973
SLIDER-555. Default ZK node is not getting deleted when you destroy an
application
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2c6cc497
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2c6cc497
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2c6cc497
Branch: refs/heads/releases/slider-0.60
Commit: 2c6cc4973edba44c5aaeea93653742e519c7188c
Parents: c7186e1
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Nov 13 18:15:03 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 13 18:24:48 2014 -0800
--
.../src/main/java/org/apache/slider/client/SliderClient.java| 5 +
1 file changed, 5 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2c6cc497/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index 164e2fe..ee5be65 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -546,6 +546,11 @@ public class SliderClient extends
AbstractSliderLaunchedService implements RunSe
if (!deleted) {
log.warn(Filesystem returned false from delete() operation);
}
+
+ if(!deleteZookeeperNode(clustername)) {
+log.warn(Unable to perform node cleanup in Zookeeper.);
+ }
+
if (fs.exists(clusterDirectory)) {
log.warn(Failed to delete {}, clusterDirectory);
}
incubator-slider git commit: SLIDER-649 extends test sleep times
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 2c6cc4973 - fb25d9356
SLIDER-649 extends test sleep times
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/fb25d935
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/fb25d935
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/fb25d935
Branch: refs/heads/releases/slider-0.60
Commit: fb25d9356353c79bd2fd89bf73b913aaa508a630
Parents: 2c6cc49
Author: Jon Maron jma...@hortonworks.com
Authored: Thu Nov 13 18:08:05 2014 -0500
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 13 18:58:10 2014 -0800
--
.../slider/agent/standalone/TestStandaloneAMRestart.groovy | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fb25d935/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMRestart.groovy
--
diff --git
a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMRestart.groovy
b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMRestart.groovy
index d8f7141..bdcf615 100644
---
a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMRestart.groovy
+++
b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAMRestart.groovy
@@ -80,7 +80,7 @@ class TestStandaloneAMRestart extends
AgentMiniClusterTestBase {
// kill again expect it to be considered a failure
killAmAndWaitForDeath(sliderClient, iteration++, clustername)
-sleep(2)
+sleep(4)
report = sliderClient.applicationReport
assert report.finalApplicationStatus == FinalApplicationStatus.FAILED
@@ -109,8 +109,8 @@ class TestStandaloneAMRestart extends
AgentMiniClusterTestBase {
iteration,
clustername)
//give yarn some time to notice
-sleep(2)
-waitUntilClusterLive(sliderClient, 2)
+sleep(4)
+waitUntilClusterLive(sliderClient, 4)
return args
}
incubator-slider git commit: SLIDER-644 Add missing license header.
Repository: incubator-slider Updated Branches: refs/heads/releases/slider-0.60 fb25d9356 - 12b0a2a5f SLIDER-644 Add missing license header. Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/12b0a2a5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/12b0a2a5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/12b0a2a5 Branch: refs/heads/releases/slider-0.60 Commit: 12b0a2a5f13418a2c10e3244a4f57a6ee1c545b4 Parents: fb25d93 Author: Josh Elser els...@apache.org Authored: Wed Nov 12 10:52:53 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Thu Nov 13 19:02:07 2014 -0800 -- .../slider/common/params/ActionDiagnosticArgs.java | 17 + 1 file changed, 17 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/12b0a2a5/slider-core/src/main/java/org/apache/slider/common/params/ActionDiagnosticArgs.java -- diff --git a/slider-core/src/main/java/org/apache/slider/common/params/ActionDiagnosticArgs.java b/slider-core/src/main/java/org/apache/slider/common/params/ActionDiagnosticArgs.java index 9feeea4..4b03336 100644 --- a/slider-core/src/main/java/org/apache/slider/common/params/ActionDiagnosticArgs.java +++ b/slider-core/src/main/java/org/apache/slider/common/params/ActionDiagnosticArgs.java @@ -1,3 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.slider.common.params; import com.beust.jcommander.Parameter;
incubator-slider git commit: Use HFile v2 for Slider HBase
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 12b0a2a5f - 98aff4d08
Use HFile v2 for Slider HBase
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/98aff4d0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/98aff4d0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/98aff4d0
Branch: refs/heads/releases/slider-0.60
Commit: 98aff4d08840530717ce29ce2cb88fbf7ec65df7
Parents: 12b0a2a
Author: tedyu yuzhih...@gmail.com
Authored: Thu Nov 13 13:12:27 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 13 19:03:31 2014 -0800
--
app-packages/hbase/appConfig-secured-default.json | 1 -
1 file changed, 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/98aff4d0/app-packages/hbase/appConfig-secured-default.json
--
diff --git a/app-packages/hbase/appConfig-secured-default.json
b/app-packages/hbase/appConfig-secured-default.json
index dc09dae..ba5ae25 100644
--- a/app-packages/hbase/appConfig-secured-default.json
+++ b/app-packages/hbase/appConfig-secured-default.json
@@ -37,7 +37,6 @@
site.hbase-site.hbase.master.info.port:
${HBASE_MASTER.ALLOCATED_PORT},
site.hbase-site.hbase.regionserver.port: 0,
site.hbase-site.hbase.master.port: 0,
-site.hbase-site.hfile.format.version: 3,
site.hbase-site.hbase.security.authentication: kerberos,
site.hbase-site.hbase.security.authorization: true,
[1/2] incubator-slider git commit: Storm should not call kinit before starting processes
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 a9deb0691 - aa1bd1945
Storm should not call kinit before starting processes
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e87ab7ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e87ab7ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e87ab7ca
Branch: refs/heads/releases/slider-0.60
Commit: e87ab7cac45b4d8d2b7664be3385666a4085c832
Parents: a9deb06
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Nov 11 12:23:10 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Nov 12 17:22:18 2014 -0800
--
app-packages/storm/package/scripts/service.py | 6 --
1 file changed, 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e87ab7ca/app-packages/storm/package/scripts/service.py
--
diff --git a/app-packages/storm/package/scripts/service.py
b/app-packages/storm/package/scripts/service.py
index fb3f83f..7a7dbdf 100644
--- a/app-packages/storm/package/scripts/service.py
+++ b/app-packages/storm/package/scripts/service.py
@@ -55,12 +55,6 @@ def service(
else:
cmd = format(env JAVA_HOME={java64_home} PATH={java64_home}/bin:$PATH
STORM_BASE_DIR={app_root} STORM_CONF_DIR={conf_dir} {python_binary} {storm_bin}
{name} {log_dir}/{name}.out 21)
-if params.security_enabled:
- if name == nimbus:
-Execute(format({kinit_path_local} -kt {storm_server_keytab_path}
{storm_jaas_server_principal}))
- else:
-Execute(format({kinit_path_local} -kt {storm_client_keytab_path}
{storm_jaas_client_principal}))
-
Execute(cmd,
not_if=no_op_test,
logoutput=False,
[2/2] incubator-slider git commit: Storm super ACL default value should be without realm
Storm super ACL default value should be without realm
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/aa1bd194
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/aa1bd194
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/aa1bd194
Branch: refs/heads/releases/slider-0.60
Commit: aa1bd1945dd99bb189f9c31b3357aab2d456ce16
Parents: e87ab7c
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Nov 11 16:08:51 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Nov 12 17:22:34 2014 -0800
--
app-packages/storm/appConfig-secured-default.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/aa1bd194/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index 21d32e5..4c40ddf 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -39,7 +39,7 @@
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
-site.storm-site.storm.zookeeper.superACL:
sasl:${USER_NAME}@EXAMPLE.COM,
+site.storm-site.storm.zookeeper.superACL: sasl:${USER_NAME},
site.storm-site.nimbus.admins: ['${USER_NAME}'],
site.storm-site.nimbus.users: ['${USER_NAME}'],
site.storm-site.nimbus.supervisor.users: ['${USER_NAME}'],
incubator-slider git commit: Storm should not call kinit before starting processes
Repository: incubator-slider
Updated Branches:
refs/heads/develop a0372d6c4 - 294f63866
Storm should not call kinit before starting processes
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/294f6386
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/294f6386
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/294f6386
Branch: refs/heads/develop
Commit: 294f638660779698dc59bd2c79d4fd66738bd51a
Parents: a0372d6
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Nov 11 12:23:10 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Nov 11 12:23:10 2014 -0800
--
app-packages/storm/package/scripts/service.py | 6 --
1 file changed, 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/294f6386/app-packages/storm/package/scripts/service.py
--
diff --git a/app-packages/storm/package/scripts/service.py
b/app-packages/storm/package/scripts/service.py
index fb3f83f..7a7dbdf 100644
--- a/app-packages/storm/package/scripts/service.py
+++ b/app-packages/storm/package/scripts/service.py
@@ -55,12 +55,6 @@ def service(
else:
cmd = format(env JAVA_HOME={java64_home} PATH={java64_home}/bin:$PATH
STORM_BASE_DIR={app_root} STORM_CONF_DIR={conf_dir} {python_binary} {storm_bin}
{name} {log_dir}/{name}.out 21)
-if params.security_enabled:
- if name == nimbus:
-Execute(format({kinit_path_local} -kt {storm_server_keytab_path}
{storm_jaas_server_principal}))
- else:
-Execute(format({kinit_path_local} -kt {storm_client_keytab_path}
{storm_jaas_client_principal}))
-
Execute(cmd,
not_if=no_op_test,
logoutput=False,
incubator-slider git commit: Storm default secured config needs a change
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 0e4c5da89 - 1ef5a95f3
Storm default secured config needs a change
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/1ef5a95f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/1ef5a95f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/1ef5a95f
Branch: refs/heads/releases/slider-0.60
Commit: 1ef5a95f37dd33cfcc2398b674fff04bb495ea35
Parents: 0e4c5da
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 10 16:31:10 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 10 16:31:10 2014 -0800
--
app-packages/storm/appConfig-secured-default.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/1ef5a95f/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index ec42acd..973c79a 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -37,7 +37,7 @@
site.storm-site.nimbus.authorizer:
backtype.storm.security.auth.authorizer.SimpleACLAuthorizer,
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
-site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.0.2.5.0-100/conf/storm_jaas.conf,
+site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
site.storm-site.storm.zookeeper.superACL:
PLACE_JAAS_CLIENT_PRINCIPAL_HERE,
site.storm-site.nimbus.admins: ['${USER_NAME}',
'PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE'],
incubator-slider git commit: Storm default secured config needs a change
Repository: incubator-slider
Updated Branches:
refs/heads/develop 4d015d209 - bedd23caf
Storm default secured config needs a change
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/bedd23ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/bedd23ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/bedd23ca
Branch: refs/heads/develop
Commit: bedd23caf555e271d748f079e05dd94804bd15d6
Parents: 4d015d2
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 10 16:31:10 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 10 16:32:25 2014 -0800
--
app-packages/storm/appConfig-secured-default.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bedd23ca/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index ec42acd..973c79a 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -37,7 +37,7 @@
site.storm-site.nimbus.authorizer:
backtype.storm.security.auth.authorizer.SimpleACLAuthorizer,
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
-site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.3.0.2.5.0-100/conf/storm_jaas.conf,
+site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
site.storm-site.storm.zookeeper.superACL:
PLACE_JAAS_CLIENT_PRINCIPAL_HERE,
site.storm-site.nimbus.admins: ['${USER_NAME}',
'PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE'],
incubator-slider git commit: Storm default secured config changes
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 1ef5a95f3 - 15faf908c
Storm default secured config changes
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/15faf908
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/15faf908
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/15faf908
Branch: refs/heads/releases/slider-0.60
Commit: 15faf908c6e16e2de53f92b896df90ab4a18f1ae
Parents: 1ef5a95
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 10 18:36:55 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 10 18:37:48 2014 -0800
--
app-packages/storm/appConfig-secured-default.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/15faf908/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index 973c79a..f0eae5c 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -22,7 +22,7 @@
site.storm-site.transactional.zookeeper.root: /transactional,
site.storm-site.storm.zookeeper.port: 2181,
site.storm-site.nimbus.childopts: -Xmx1024m
-Djava.security.auth.login.config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${@//site/global/ganglia_server_host},port=${@//site/global/ganglia_server_port},wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM,
-site.storm-site.supervisor.childopts: -Xmx256m
-Djava.security.auth.login.config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}conf/storm_jaas.conf
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${@//site/global/ganglia_server_host},port=${@//site/global/ganglia_server_port},wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM,
+site.storm-site.supervisor.childopts: -Xmx256m
-Djava.security.auth.login.config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${@//site/global/ganglia_server_host},port=${@//site/global/ganglia_server_port},wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM,
site.storm-site.ui.childopts: -Xmx768m
-Djava.security.auth.login.config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf,
site.storm-site.worker.childopts: -Xmx768m
-javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${@//site/global/ganglia_server_host},port=${@//site/global/ganglia_server_port},wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/external/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM,
site.storm-site.dev.zookeeper.path:
${AGENT_WORK_ROOT}/app/tmp/dev-storm-zookeeper,
incubator-slider git commit: Storm secured config defaults for principal names
Repository: incubator-slider
Updated Branches:
refs/heads/develop 68263b4ed - d608dbae4
Storm secured config defaults for principal names
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/d608dbae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/d608dbae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/d608dbae
Branch: refs/heads/develop
Commit: d608dbae4d667f6942a5d9fd34186bdcc12073a8
Parents: 68263b4
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 10 22:34:28 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 10 22:34:28 2014 -0800
--
app-packages/storm/appConfig-secured-default.json | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d608dbae/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index f0eae5c..21d32e5 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -39,10 +39,10 @@
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
-site.storm-site.storm.zookeeper.superACL:
PLACE_JAAS_CLIENT_PRINCIPAL_HERE,
-site.storm-site.nimbus.admins: ['${USER_NAME}',
'PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE'],
+site.storm-site.storm.zookeeper.superACL:
sasl:${USER_NAME}@EXAMPLE.COM,
+site.storm-site.nimbus.admins: ['${USER_NAME}'],
site.storm-site.nimbus.users: ['${USER_NAME}'],
-site.storm-site.nimbus.supervisor.users:
['PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE'],
+site.storm-site.nimbus.supervisor.users: ['${USER_NAME}'],
site.storm-site.nimubs.authorizer:
backtype.storm.security.auth.authorizer.SimpleACLAuthorizer,
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
@@ -50,8 +50,8 @@
site.storm-site.ui.filter.params: {'type': 'kerberos',
'kerberos.principal': 'HTTP/_HOST', 'kerberos.keytab':
'/etc/security/keytabs/spnego.service.keytab', 'kerberos.name.rules':
'RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT'},
site.storm-env.kerberos_domain: EXAMPLE.COM,
-site.storm-env.storm_client_principal_name:
PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE,
-site.storm-env.storm_server_principal_name:
PLACE_JAAS_STORMSERVER_PRINCIPAL_HERE/_HOST,
+site.storm-env.storm_client_principal_name: ${USER_NAME}@EXAMPLE.COM,
+site.storm-env.storm_server_principal_name:
${USER_NAME}/_h...@example.com,
site.storm-env.storm_client_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.client.keytab,
site.storm-env.storm_server_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.nimbus.keytab
[2/2] incubator-slider git commit: Storm secured config defaults for principal names
Storm secured config defaults for principal names
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/71a8559d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/71a8559d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/71a8559d
Branch: refs/heads/releases/slider-0.60
Commit: 71a8559da2e80b9581ca938e5433d9313c60e8c8
Parents: a26840e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 10 22:34:28 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 10 22:47:23 2014 -0800
--
app-packages/storm/appConfig-secured-default.json | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/71a8559d/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index f0eae5c..21d32e5 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -39,10 +39,10 @@
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
site.storm-site.java.security.auth.login.config:
${AGENT_WORK_ROOT}/app/install/apache-storm-${pkg.version}/conf/storm_jaas.conf,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
-site.storm-site.storm.zookeeper.superACL:
PLACE_JAAS_CLIENT_PRINCIPAL_HERE,
-site.storm-site.nimbus.admins: ['${USER_NAME}',
'PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE'],
+site.storm-site.storm.zookeeper.superACL:
sasl:${USER_NAME}@EXAMPLE.COM,
+site.storm-site.nimbus.admins: ['${USER_NAME}'],
site.storm-site.nimbus.users: ['${USER_NAME}'],
-site.storm-site.nimbus.supervisor.users:
['PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE'],
+site.storm-site.nimbus.supervisor.users: ['${USER_NAME}'],
site.storm-site.nimubs.authorizer:
backtype.storm.security.auth.authorizer.SimpleACLAuthorizer,
site.storm-site.storm.thrift.transport:
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin,
site.storm-site.storm.principal.tolocal:
backtype.storm.security.auth.KerberosPrincipalToLocal,
@@ -50,8 +50,8 @@
site.storm-site.ui.filter.params: {'type': 'kerberos',
'kerberos.principal': 'HTTP/_HOST', 'kerberos.keytab':
'/etc/security/keytabs/spnego.service.keytab', 'kerberos.name.rules':
'RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT'},
site.storm-env.kerberos_domain: EXAMPLE.COM,
-site.storm-env.storm_client_principal_name:
PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE,
-site.storm-env.storm_server_principal_name:
PLACE_JAAS_STORMSERVER_PRINCIPAL_HERE/_HOST,
+site.storm-env.storm_client_principal_name: ${USER_NAME}@EXAMPLE.COM,
+site.storm-env.storm_server_principal_name:
${USER_NAME}/_h...@example.com,
site.storm-env.storm_client_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.client.keytab,
site.storm-env.storm_server_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.nimbus.keytab
[1/2] incubator-slider git commit: SLIDER-634. slider wrapper should look for and use python version greater than or equal to 2.6 (Yu Liu via smohanty)
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 5591b7209 - 71a8559da
SLIDER-634. slider wrapper should look for and use python version greater than
or equal to 2.6 (Yu Liu via smohanty)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a26840e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a26840e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a26840e3
Branch: refs/heads/releases/slider-0.60
Commit: a26840e3e73f12e40a8d6c468b60932f654c160d
Parents: 5591b72
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 10 19:47:53 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 10 22:47:10 2014 -0800
--
slider-assembly/src/main/scripts/slider | 14 --
1 file changed, 12 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a26840e3/slider-assembly/src/main/scripts/slider
--
diff --git a/slider-assembly/src/main/scripts/slider
b/slider-assembly/src/main/scripts/slider
index 9646694..97dabaf 100755
--- a/slider-assembly/src/main/scripts/slider
+++ b/slider-assembly/src/main/scripts/slider
@@ -23,11 +23,21 @@ import os
import sys
import subprocess
-ON_POSIX = 'posix' in sys.builtin_module_names
+pyExecPath = None
+
+if os.path.isfile('/usr/bin/python2.7') and (pyExecPath is None):
+ pyExecPath = '/usr/bin/python2.7'
+if os.path.isfile('/usr/bin/python2.6') and (pyExecPath is None):
+ pyExecPath = '/usr/bin/python2.6'
+if os.path.isfile('/usr/bin/python') and (pyExecPath is None):
+ pyExecPath = '/usr/bin/python'
+if pyExecPath is None:
+ pyExecPath = 'python'
+ON_POSIX = 'posix' in sys.builtin_module_names
currDir = os.path.dirname(os.path.realpath(__file__))
args = sys.argv[1:]
-cmd = [currDir + /slider.py]
+cmd = [pyExecPath + + currDir + /slider.py]
cmd.extend(args)
finalCmd = .join(cmd)
Git Push Summary
Repository: incubator-slider Updated Branches: refs/heads/feature/python_unittests [created] b4112d241
incubator-slider git commit: Test Commit 3
Repository: incubator-slider Updated Branches: refs/heads/feature/python_unittests 0cc2a39fb - 52591a9fa Test Commit 3 Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/52591a9f Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/52591a9f Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/52591a9f Branch: refs/heads/feature/python_unittests Commit: 52591a9fabdedd7fc0058d7cdf7c8c826dfda50c Parents: 0cc2a39 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 07:56:08 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 07:56:08 2014 -0800 -- slider-agent/src/test/python/agent/TestMain.py | 32 .../TestDirectoryResource.py| 14 +++-- .../resource_management/TestExecuteResource.py | 15 - .../resource_management/TestFileResource.py | 20 ++-- 4 files changed, 33 insertions(+), 48 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/52591a9f/slider-agent/src/test/python/agent/TestMain.py -- diff --git a/slider-agent/src/test/python/agent/TestMain.py b/slider-agent/src/test/python/agent/TestMain.py index 7c0036b..256440e 100644 --- a/slider-agent/src/test/python/agent/TestMain.py +++ b/slider-agent/src/test/python/agent/TestMain.py @@ -33,6 +33,9 @@ import tempfile from Controller import Controller from Registry import Registry from optparse import OptionParser +import platform + +IS_WINDOWS = platform.system() == Windows logger = logging.getLogger() @@ -116,15 +119,15 @@ class TestMain(unittest.TestCase): main.update_log_level(config, tmpoutfile) setLevel_mock.assert_called_with(logging.INFO) - - @patch(signal.signal) - def test_bind_signal_handlers(self, signal_mock): -main.bind_signal_handlers() -# Check if on SIGINT/SIGTERM agent is configured to terminate -signal_mock.assert_any_call(signal.SIGINT, main.signal_handler) -signal_mock.assert_any_call(signal.SIGTERM, main.signal_handler) -# Check if on SIGUSR1 agent is configured to fall into debug -signal_mock.assert_any_call(signal.SIGUSR1, main.debug) + if not IS_WINDOWS: +@patch(signal.signal) +def test_bind_signal_handlers(self, signal_mock): + main.bind_signal_handlers() + # Check if on SIGINT/SIGTERM agent is configured to terminate + signal_mock.assert_any_call(signal.SIGINT, main.signal_handler) + signal_mock.assert_any_call(signal.SIGTERM, main.signal_handler) + # Check if on SIGUSR1 agent is configured to fall into debug + signal_mock.assert_any_call(signal.SIGUSR1, main.debug) @patch(os.path.exists) @@ -201,7 +204,8 @@ class TestMain(unittest.TestCase): # Testing normal exit exists_mock.return_value = False main.stop_agent() -kill_mock.assert_called_with(int(pid), signal.SIGTERM) +if not IS_WINDOWS: + kill_mock.assert_called_with(int(pid), signal.SIGTERM) # Restore kill_mock.reset_mock() @@ -210,8 +214,9 @@ class TestMain(unittest.TestCase): # Testing exit when failed to remove pid file exists_mock.return_value = True main.stop_agent() -kill_mock.assert_any_call(int(pid), signal.SIGTERM) -kill_mock.assert_any_call(int(pid), signal.SIGKILL) +if not IS_WINDOWS: + kill_mock.assert_any_call(int(pid), signal.SIGTERM) + kill_mock.assert_any_call(int(pid), signal.SIGKILL) _exit_mock.assert_called_with(1) # Restore @@ -253,7 +258,8 @@ class TestMain(unittest.TestCase): main.main() self.assertTrue(setup_logging_mock.called) -self.assertTrue(bind_signal_handlers_mock.called) +if not IS_WINDOWS: + self.assertTrue(bind_signal_handlers_mock.called) self.assertTrue(update_config_from_file_mock.called) self.assertTrue(perform_prestart_checks_mock.called) self.assertTrue(write_pid_mock.called) http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/52591a9f/slider-agent/src/test/python/resource_management/TestDirectoryResource.py -- diff --git a/slider-agent/src/test/python/resource_management/TestDirectoryResource.py b/slider-agent/src/test/python/resource_management/TestDirectoryResource.py index 0b74708..e3282d0 100644 --- a/slider-agent/src/test/python/resource_management/TestDirectoryResource.py +++ b/slider-agent/src/test/python/resource_management/TestDirectoryResource.py @@ -33,11 +33,10 @@ class TestDirectoryResource(TestCase): @patch.object(os.path, isdir) @patch.object(os, stat) @patch.object(os,chmod) - @patch.object(os,chown)
incubator-slider git commit: Test Commit 5
Repository: incubator-slider Updated Branches: refs/heads/feature/python_unittests 4c21c63a3 - 8bc4ea989 Test Commit 5 Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/8bc4ea98 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/8bc4ea98 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/8bc4ea98 Branch: refs/heads/feature/python_unittests Commit: 8bc4ea989737171349f44af8056319b159962765 Parents: 4c21c63 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 08:26:53 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 08:26:53 2014 -0800 -- slider-agent/src/test/python/agent/TestMain.py | 62 ++-- .../resource_management/TestContentSources.py | 14 - 2 files changed, 42 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8bc4ea98/slider-agent/src/test/python/agent/TestMain.py -- diff --git a/slider-agent/src/test/python/agent/TestMain.py b/slider-agent/src/test/python/agent/TestMain.py index 256440e..537ab1f 100644 --- a/slider-agent/src/test/python/agent/TestMain.py +++ b/slider-agent/src/test/python/agent/TestMain.py @@ -183,45 +183,43 @@ class TestMain(unittest.TestCase): main.perform_prestart_checks(main.config) self.assertFalse(exit_mock.called) - - @patch(time.sleep) - @patch(os.kill) - @patch(os._exit) - @patch(os.path.exists) - def test_daemonize_and_stop(self, exists_mock, _exit_mock, kill_mock, + if not IS_WINDOWS: +@patch(time.sleep) +@patch(os.kill) +@patch(os._exit) +@patch(os.path.exists) +def test_daemonize_and_stop(self, exists_mock, _exit_mock, kill_mock, sleep_mock): -oldpid = ProcessHelper.pidfile -pid = str(os.getpid()) -_, tmpoutfile = tempfile.mkstemp() -ProcessHelper.pidfile = tmpoutfile - -# Test daemonization -main.write_pid() -saved = open(ProcessHelper.pidfile, 'r').read() -self.assertEqual(pid, saved) - -# Reuse pid file when testing agent stop -# Testing normal exit -exists_mock.return_value = False -main.stop_agent() -if not IS_WINDOWS: + oldpid = ProcessHelper.pidfile + pid = str(os.getpid()) + _, tmpoutfile = tempfile.mkstemp() + ProcessHelper.pidfile = tmpoutfile + + # Test daemonization + main.write_pid() + saved = open(ProcessHelper.pidfile, 'r').read() + self.assertEqual(pid, saved) + + # Reuse pid file when testing agent stop + # Testing normal exit + exists_mock.return_value = False + main.stop_agent() kill_mock.assert_called_with(int(pid), signal.SIGTERM) -# Restore -kill_mock.reset_mock() -_exit_mock.reset_mock() + # Restore + kill_mock.reset_mock() + _exit_mock.reset_mock() -# Testing exit when failed to remove pid file -exists_mock.return_value = True -main.stop_agent() -if not IS_WINDOWS: + # Testing exit when failed to remove pid file + exists_mock.return_value = True + main.stop_agent() kill_mock.assert_any_call(int(pid), signal.SIGTERM) kill_mock.assert_any_call(int(pid), signal.SIGKILL) -_exit_mock.assert_called_with(1) + _exit_mock.assert_called_with(1) -# Restore -ProcessHelper.pidfile = oldpid -os.remove(tmpoutfile) + # Restore + ProcessHelper.pidfile = oldpid + os.remove(tmpoutfile) @patch.object(Registry, readAMHostPort) @patch.object(main, setup_logging) http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/8bc4ea98/slider-agent/src/test/python/resource_management/TestContentSources.py -- diff --git a/slider-agent/src/test/python/resource_management/TestContentSources.py b/slider-agent/src/test/python/resource_management/TestContentSources.py index 3913ba1..7015c47 100644 --- a/slider-agent/src/test/python/resource_management/TestContentSources.py +++ b/slider-agent/src/test/python/resource_management/TestContentSources.py @@ -30,6 +30,9 @@ from resource_management.core.source import InlineTemplate from jinja2 import UndefinedError, TemplateNotFound import urllib2 import os +import platform + +IS_WINDOWS = platform.system() == Windows @patch.object(System, os_family, new = 'redhat') @@ -46,8 +49,12 @@ class TestContentSources(TestCase): file_mock.read.return_value = 'content' open_mock.return_value = file_mock +filepath = /absolute/path/file +if IS_WINDOWS: + filepath = \\absolute\\path\\file + with Environment(/base) as env: - static_file = StaticFile(/absolute/path/file) + static_file
incubator-slider git commit: Test Commit 8
Repository: incubator-slider
Updated Branches:
refs/heads/feature/python_unittests 1425f76a0 - 801825d0a
Test Commit 8
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/801825d0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/801825d0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/801825d0
Branch: refs/heads/feature/python_unittests
Commit: 801825d0a976f4a0b7c341386f9fa438817a8966
Parents: 1425f76
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Nov 9 09:15:14 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Nov 9 09:15:14 2014 -0800
--
.../src/main/python/agent/PythonExecutor.py | 24 ++--
1 file changed, 12 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/801825d0/slider-agent/src/main/python/agent/PythonExecutor.py
--
diff --git a/slider-agent/src/main/python/agent/PythonExecutor.py
b/slider-agent/src/main/python/agent/PythonExecutor.py
index 866755f..97e8b1f 100644
--- a/slider-agent/src/main/python/agent/PythonExecutor.py
+++ b/slider-agent/src/main/python/agent/PythonExecutor.py
@@ -101,18 +101,18 @@ class PythonExecutor:
out = open(tmpoutfile, 'r').read()
error = open(tmperrfile, 'r').read()
-try:
- with open(tmpstructedoutfile, 'r') as fp:
-structured_out = json.load(fp)
-except Exception:
- if os.path.exists(tmpstructedoutfile):
-errMsg = 'Unable to read structured output from ' + tmpstructedoutfile
-structured_out = {
- 'msg': errMsg
-}
-logger.warn(structured_out)
- else:
-structured_out = {}
+structured_out = {}
+if os.path.exists(tmpstructedoutfile):
+ try:
+with open(tmpstructedoutfile, 'r') as fp:
+ structured_out = json.load(fp)
+ except Exception as e:
+if os.path.exists(tmpstructedoutfile):
+ errMsg = 'Unable to read structured output from ' +
tmpstructedoutfile + ' ' + str(e)
+ structured_out = {
+'msg': errMsg
+ }
+ logger.warn(structured_out)
if self.python_process_has_been_killed:
error = str(error) + \n Python script has been killed due to timeout
[1/2] incubator-slider git commit: Test Commit 9
Repository: incubator-slider Updated Branches: refs/heads/feature/python_unittests 801825d0a - ea1bd184f Test Commit 9 Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/58f4c8b3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/58f4c8b3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/58f4c8b3 Branch: refs/heads/feature/python_unittests Commit: 58f4c8b3c35754a2f805876ca8b3bbf36efcf722 Parents: 801825d Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 09:57:42 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 09:57:42 2014 -0800 -- slider-agent/src/test/python/agent/TestPythonExecutor.py | 3 +++ 1 file changed, 3 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58f4c8b3/slider-agent/src/test/python/agent/TestPythonExecutor.py -- diff --git a/slider-agent/src/test/python/agent/TestPythonExecutor.py b/slider-agent/src/test/python/agent/TestPythonExecutor.py index 6506758..e6a8592 100644 --- a/slider-agent/src/test/python/agent/TestPythonExecutor.py +++ b/slider-agent/src/test/python/agent/TestPythonExecutor.py @@ -125,6 +125,9 @@ class TestPythonExecutor(TestCase): _, tmpoutfile = tempfile.mkstemp() _, tmperrfile = tempfile.mkstemp() _, tmpstroutfile = tempfile.mkstemp() +if IS_WINDOWS: + if os.path.exists(tmpstroutfile): +os.remove(tmpstroutfile) PYTHON_TIMEOUT_SECONDS = 5 def launch_python_subprocess_method(command, tmpout, tmperr, environment_vars):
[2/2] incubator-slider git commit: Test Commit 10
Test Commit 10 Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/ea1bd184 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/ea1bd184 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/ea1bd184 Branch: refs/heads/feature/python_unittests Commit: ea1bd184f0717231604f584325fd8cd485971375 Parents: 58f4c8b Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 10:02:36 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 10:02:36 2014 -0800 -- slider-agent/src/test/python/agent/TestPythonExecutor.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/ea1bd184/slider-agent/src/test/python/agent/TestPythonExecutor.py -- diff --git a/slider-agent/src/test/python/agent/TestPythonExecutor.py b/slider-agent/src/test/python/agent/TestPythonExecutor.py index e6a8592..9a46bd5 100644 --- a/slider-agent/src/test/python/agent/TestPythonExecutor.py +++ b/slider-agent/src/test/python/agent/TestPythonExecutor.py @@ -127,7 +127,7 @@ class TestPythonExecutor(TestCase): _, tmpstroutfile = tempfile.mkstemp() if IS_WINDOWS: if os.path.exists(tmpstroutfile): -os.remove(tmpstroutfile) +tmpstroutfile = tmpstroutfile + _t PYTHON_TIMEOUT_SECONDS = 5 def launch_python_subprocess_method(command, tmpout, tmperr, environment_vars):
incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Repository: incubator-slider
Updated Branches:
refs/heads/develop b4112d241 - b8fbf088f
SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b8fbf088
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b8fbf088
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b8fbf088
Branch: refs/heads/develop
Commit: b8fbf088fe48ed405f8fb43fb92aca7735049618
Parents: b4112d2
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Nov 9 10:32:22 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Nov 9 10:32:22 2014 -0800
--
slider-agent/pom.xml| 28 +++-
.../src/main/python/agent/PythonExecutor.py | 7 +-
slider-agent/src/main/python/agent/main.py | 3 +-
.../src/test/python/agent/TestActionQueue.py| 15 ++
.../agent/TestCustomServiceOrchestrator.py | 12 +-
slider-agent/src/test/python/agent/TestGrep.py | 13 +-
slider-agent/src/test/python/agent/TestMain.py | 96 ++--
.../src/test/python/agent/TestPythonExecutor.py | 28 +++-
.../src/test/python/agent/TestRegistration.py | 11 +-
.../resource_management/TestContentSources.py | 148 +++
.../resource_management/TestCopyFromLocal.py| 65
.../TestDirectoryResource.py| 14 +-
.../resource_management/TestExecuteResource.py | 92 +++-
.../resource_management/TestFileResource.py | 20 +--
.../resource_management/TestLinkResource.py | 148 ---
.../TestPropertiesFileResource.py | 5 +-
16 files changed, 224 insertions(+), 481 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b8fbf088/slider-agent/pom.xml
--
diff --git a/slider-agent/pom.xml b/slider-agent/pom.xml
index bcfc209..f4d50b2 100644
--- a/slider-agent/pom.xml
+++ b/slider-agent/pom.xml
@@ -32,8 +32,9 @@
package.release1/package.release
skipTestsfalse/skipTests
python.verpython gt;= 2.6/python.ver
+
executable.python${project.basedir}/../slider-agent/src/test/python/python-wrap/executable.python
+
python.path.l${project.basedir}/src/main/python/jinja2:${project.basedir}/src/test/python:${project.basedir}/src/main/python:${project.basedir}/src/main/python/agent:${project.basedir}/src/main/python/resource_management:${project.basedir}/src/test/python/agent:${project.basedir}/src/test/python/resource_management:${project.basedir}/src/main/python/kazoo/python.path.l
/properties
-
build
plugins
@@ -63,13 +64,13 @@
executions
execution
configuration
-
executable${project.basedir}/../slider-agent/src/test/python/python-wrap/executable
+ executable${executable.python}/executable
workingDirectorysrc/test/python/workingDirectory
arguments
argumentunitTests.py/argument
/arguments
environmentVariables
-
PYTHONPATH${project.basedir}/src/main/python/jinja2:${project.basedir}/src/test/python:${project.basedir}/src/main/python:${project.basedir}/src/main/python/agent:${project.basedir}/src/main/python/resource_management:${project.basedir}/src/test/python/agent:${project.basedir}/src/test/python/resource_management:${project.basedir}/src/main/python/kazoo/PYTHONPATH
+PYTHONPATH${python.path.l}/PYTHONPATH
/environmentVariables
skip${skipTests}/skip
/configuration
@@ -94,6 +95,27 @@
/build
profiles
+ profile
+ idWindows/id
+ activation
+osfamilywindows/family/os
+ /activation
+ properties
+executable.pythonpython/executable.python
+
python.path.l${project.basedir}\src\main\python\jinja2;${project.basedir}\src\test\python;${project.basedir}\src\main\python;${project.basedir}\src\main\python\agent;${project.basedir}\src\main\python\resource_management;${project.basedir}\src\test\python\agent;${project.basedir}\src\test\python\resource_management;${project.basedir}\src\main\python\kazoo/python.path.l
+ /properties
+/profile
+
+profile
+ idLinux/id
+ activation
+osfamily!windows/family/os
+ /activation
+ properties
+
executable.python${project.basedir}/../slider-agent/src/test/python/python-wrap/executable.python
+
incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 5cdaa7d42 - d6299b746
SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/d6299b74
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/d6299b74
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/d6299b74
Branch: refs/heads/releases/slider-0.60
Commit: d6299b746668828a048a19e04c5a8c326bd9666e
Parents: 5cdaa7d
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Nov 9 10:32:22 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Nov 9 11:49:05 2014 -0800
--
slider-agent/pom.xml| 28 +++-
.../src/main/python/agent/PythonExecutor.py | 7 +-
slider-agent/src/main/python/agent/main.py | 3 +-
.../src/test/python/agent/TestActionQueue.py| 15 ++
.../agent/TestCustomServiceOrchestrator.py | 12 +-
slider-agent/src/test/python/agent/TestGrep.py | 13 +-
slider-agent/src/test/python/agent/TestMain.py | 96 ++--
.../src/test/python/agent/TestPythonExecutor.py | 28 +++-
.../src/test/python/agent/TestRegistration.py | 11 +-
.../resource_management/TestContentSources.py | 148 +++
.../resource_management/TestCopyFromLocal.py| 65
.../TestDirectoryResource.py| 14 +-
.../resource_management/TestExecuteResource.py | 92 +++-
.../resource_management/TestFileResource.py | 20 +--
.../resource_management/TestLinkResource.py | 148 ---
.../TestPropertiesFileResource.py | 5 +-
16 files changed, 224 insertions(+), 481 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d6299b74/slider-agent/pom.xml
--
diff --git a/slider-agent/pom.xml b/slider-agent/pom.xml
index 4e5b6bd..1bbea9d 100644
--- a/slider-agent/pom.xml
+++ b/slider-agent/pom.xml
@@ -32,8 +32,9 @@
package.release1/package.release
skipTestsfalse/skipTests
python.verpython gt;= 2.6/python.ver
+
executable.python${project.basedir}/../slider-agent/src/test/python/python-wrap/executable.python
+
python.path.l${project.basedir}/src/main/python/jinja2:${project.basedir}/src/test/python:${project.basedir}/src/main/python:${project.basedir}/src/main/python/agent:${project.basedir}/src/main/python/resource_management:${project.basedir}/src/test/python/agent:${project.basedir}/src/test/python/resource_management:${project.basedir}/src/main/python/kazoo/python.path.l
/properties
-
build
plugins
@@ -63,13 +64,13 @@
executions
execution
configuration
-
executable${project.basedir}/../slider-agent/src/test/python/python-wrap/executable
+ executable${executable.python}/executable
workingDirectorysrc/test/python/workingDirectory
arguments
argumentunitTests.py/argument
/arguments
environmentVariables
-
PYTHONPATH${project.basedir}/src/main/python/jinja2:${project.basedir}/src/test/python:${project.basedir}/src/main/python:${project.basedir}/src/main/python/agent:${project.basedir}/src/main/python/resource_management:${project.basedir}/src/test/python/agent:${project.basedir}/src/test/python/resource_management:${project.basedir}/src/main/python/kazoo/PYTHONPATH
+PYTHONPATH${python.path.l}/PYTHONPATH
/environmentVariables
skip${skipTests}/skip
/configuration
@@ -94,6 +95,27 @@
/build
profiles
+ profile
+ idWindows/id
+ activation
+osfamilywindows/family/os
+ /activation
+ properties
+executable.pythonpython/executable.python
+
python.path.l${project.basedir}\src\main\python\jinja2;${project.basedir}\src\test\python;${project.basedir}\src\main\python;${project.basedir}\src\main\python\agent;${project.basedir}\src\main\python\resource_management;${project.basedir}\src\test\python\agent;${project.basedir}\src\test\python\resource_management;${project.basedir}\src\main\python\kazoo/python.path.l
+ /properties
+/profile
+
+profile
+ idLinux/id
+ activation
+osfamily!windows/family/os
+ /activation
+ properties
+
executable.python${project.basedir}/../slider-agent/src/test/python/python-wrap/executable.python
+
incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Repository: incubator-slider
Updated Branches:
refs/heads/develop b8fbf088f - 22be64644
SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/22be6464
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/22be6464
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/22be6464
Branch: refs/heads/develop
Commit: 22be64644e3e622e8944f21f10d2387b3651a0f1
Parents: b8fbf08
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Nov 9 14:26:54 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Nov 9 14:26:54 2014 -0800
--
slider-agent/src/test/python/python-wrap| 13 +
.../resource_management/TestExecuteResource.py | 2 +-
slider-agent/src/test/python/unitTests.py | 2 +-
slider-assembly/pom.xml | 28 ++--
slider-assembly/src/test/python/unitTests.py| 2 +-
5 files changed, 37 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/22be6464/slider-agent/src/test/python/python-wrap
--
diff --git a/slider-agent/src/test/python/python-wrap
b/slider-agent/src/test/python/python-wrap
index ea57721..88a8c55 100755
--- a/slider-agent/src/test/python/python-wrap
+++ b/slider-agent/src/test/python/python-wrap
@@ -17,23 +17,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-export PYTHONPATH=/usr/lib/python2.6/site-packages:$PYTHONPATH
-
# reset settings
unset PYTHON
+if [ -a /usr/bin/python2.6 ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python2.6
+fi
+
# checking for preferable python versions
if [ -a /usr/bin/python2.7 ] [ -z $PYTHON ]; then
PYTHON=/usr/bin/python2.7
fi
-if [ -a /usr/bin/python2.6 ] [ -z $PYTHON ]; then
- PYTHON=/usr/bin/python2.6
+# if no preferable python versions found, try to use system one
+if [ -a /usr/bin/python ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python
fi
# if no preferable python versions found, try to use system one
if [[ -z $PYTHON ]]; then
- PYTHON=/usr/bin/python
+ PYTHON=python
fi
# execute script
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/22be6464/slider-agent/src/test/python/resource_management/TestExecuteResource.py
--
diff --git
a/slider-agent/src/test/python/resource_management/TestExecuteResource.py
b/slider-agent/src/test/python/resource_management/TestExecuteResource.py
index f7f6371..0673b66 100644
--- a/slider-agent/src/test/python/resource_management/TestExecuteResource.py
+++ b/slider-agent/src/test/python/resource_management/TestExecuteResource.py
@@ -77,7 +77,7 @@ class TestExecuteResource(TestCase):
poll_after = 5)
self.assertTrue(False, Should fail as process does not run for 5
seconds)
except Fail as e:
-self.assertTrue(returned 1 in e.message)
+self.assertTrue(returned 1 in str(e))
pass
self.assertTrue(popen_mock.called, 'subprocess.Popen should have been
called!')
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/22be6464/slider-agent/src/test/python/unitTests.py
--
diff --git a/slider-agent/src/test/python/unitTests.py
b/slider-agent/src/test/python/unitTests.py
index 6aa0167..b65c075 100644
--- a/slider-agent/src/test/python/unitTests.py
+++ b/slider-agent/src/test/python/unitTests.py
@@ -32,7 +32,7 @@ ignoredDirs = [mock]
class TestAgent(unittest.TestSuite):
def run(self, result, debug=False):
run = unittest.TestSuite.run
-run(self, result, debug)
+run(self, result)
return result
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/22be6464/slider-assembly/pom.xml
--
diff --git a/slider-assembly/pom.xml b/slider-assembly/pom.xml
index f549b36..88c3693 100644
--- a/slider-assembly/pom.xml
+++ b/slider-assembly/pom.xml
@@ -38,6 +38,9 @@
src.confdirsrc/conf-hdp/src.confdir
src.libdir${project.build.directory}/lib/src.libdir
src.agent.ini.dir${project.build.directory}/../../slider-agent/conf/src.agent.ini.dir
+python.verpython gt;= 2.6/python.ver
+
executable.python${project.basedir}/../slider-agent/src/test/python/python-wrap/executable.python
+
python.path.l${project.basedir}/src/main/scripts:${project.basedir}/../slider-agent/src/test/python/mock:${project.basedir}/src/test/python/scripts/python.path.l
skipTestsfalse/skipTests
/properties
@@ -72,13 +75,13 @@
incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Repository: incubator-slider Updated Branches: refs/heads/develop 22be64644 - cc9d0d063 SLIDER-630. slider-agent unit tests fail on debian and suse and windows Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/cc9d0d06 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/cc9d0d06 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/cc9d0d06 Branch: refs/heads/develop Commit: cc9d0d063278967efdb4da50c8bff302b2752313 Parents: 22be646 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 14:34:38 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 14:34:38 2014 -0800 -- slider-agent/src/test/python/agent/TestPythonExecutor.py | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/cc9d0d06/slider-agent/src/test/python/agent/TestPythonExecutor.py -- diff --git a/slider-agent/src/test/python/agent/TestPythonExecutor.py b/slider-agent/src/test/python/agent/TestPythonExecutor.py index 9a46bd5..0a43639 100644 --- a/slider-agent/src/test/python/agent/TestPythonExecutor.py +++ b/slider-agent/src/test/python/agent/TestPythonExecutor.py @@ -33,6 +33,7 @@ from PythonExecutor import PythonExecutor from AgentConfig import AgentConfig from mock.mock import MagicMock, patch from AgentToggleLogger import AgentToggleLogger +import os class TestPythonExecutor(TestCase): def setUp(self):
incubator-slider git commit: SLIDER-630. slider-assembly unit tests fail on debian and suse and windows
Repository: incubator-slider Updated Branches: refs/heads/develop cc9d0d063 - 1890c427b SLIDER-630. slider-assembly unit tests fail on debian and suse and windows Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/1890c427 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/1890c427 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/1890c427 Branch: refs/heads/develop Commit: 1890c427b881934e80f1d426a8bc6b5643fd878b Parents: cc9d0d0 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 14:56:19 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 14:56:19 2014 -0800 -- .../src/test/python/scripts/TestSlider.py | 24 +++- 1 file changed, 18 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/1890c427/slider-assembly/src/test/python/scripts/TestSlider.py -- diff --git a/slider-assembly/src/test/python/scripts/TestSlider.py b/slider-assembly/src/test/python/scripts/TestSlider.py index 6fe2a47..0ed5386 100644 --- a/slider-assembly/src/test/python/scripts/TestSlider.py +++ b/slider-assembly/src/test/python/scripts/TestSlider.py @@ -25,28 +25,40 @@ import unittest import logging import slider import os +import platform + +IS_WINDOWS = platform.system() == Windows logger = logging.getLogger() class TestSlider(unittest.TestCase): + @patch(os.environ.get) @patch.object(slider, confDir) @patch.object(slider, libDir) @patch.object(slider, executeEnvSh) @patch(os.path.exists) @patch.object(slider, java) - def test_main(self, java_mock, exists_mock, executeEnvSh_mock, libDir_mock, confDir_mock): + def test_main(self, java_mock, exists_mock, executeEnvSh_mock, libDir_mock, confDir_mock, os_env_get_mock): sys.argv = [slider, list] exists_mock.return_value = True libDir_mock.return_value = /dir/libdir confDir_mock.return_value = /dir/confdir +os_env_get_mock.return_value = env_val slider.main() self.assertTrue(java_mock.called) -java_mock.assert_called_with( - 'org.apache.slider.Slider', - ['list'], - '/dir/libdir/*:/dir/confdir::', - ['-Dslider.confdir=/dir/confdir', '-Dslider.libdir=/dir/libdir', '-Djava.net.preferIPv4Stack=true', '-Djava.awt.headless=true', '-Xmx256m']) +if IS_WINDOWS: + java_mock.assert_called_with( +'org.apache.slider.Slider', +['list'], +'/dir/libdir\\*;/dir/confdir;env_val;env_val', +['-Dslider.confdir=/dir/confdir', '-Dslider.libdir=/dir/libdir', 'env_val']) +else: + java_mock.assert_called_with( +'org.apache.slider.Slider', +['list'], +'/dir/libdir/*:/dir/confdir:env_val:env_val', +['-Dslider.confdir=/dir/confdir', '-Dslider.libdir=/dir/libdir', 'env_val']) pass
[3/3] incubator-slider git commit: SLIDER-630. slider-assembly unit tests fail on debian and suse and windows
SLIDER-630. slider-assembly unit tests fail on debian and suse and windows Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a036ec11 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a036ec11 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a036ec11 Branch: refs/heads/releases/slider-0.60 Commit: a036ec112331fa33aebcb5df0464579d2ae4d6b6 Parents: 131221e Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 14:56:19 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 15:07:25 2014 -0800 -- .../src/test/python/scripts/TestSlider.py | 24 +++- 1 file changed, 18 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a036ec11/slider-assembly/src/test/python/scripts/TestSlider.py -- diff --git a/slider-assembly/src/test/python/scripts/TestSlider.py b/slider-assembly/src/test/python/scripts/TestSlider.py index 6fe2a47..0ed5386 100644 --- a/slider-assembly/src/test/python/scripts/TestSlider.py +++ b/slider-assembly/src/test/python/scripts/TestSlider.py @@ -25,28 +25,40 @@ import unittest import logging import slider import os +import platform + +IS_WINDOWS = platform.system() == Windows logger = logging.getLogger() class TestSlider(unittest.TestCase): + @patch(os.environ.get) @patch.object(slider, confDir) @patch.object(slider, libDir) @patch.object(slider, executeEnvSh) @patch(os.path.exists) @patch.object(slider, java) - def test_main(self, java_mock, exists_mock, executeEnvSh_mock, libDir_mock, confDir_mock): + def test_main(self, java_mock, exists_mock, executeEnvSh_mock, libDir_mock, confDir_mock, os_env_get_mock): sys.argv = [slider, list] exists_mock.return_value = True libDir_mock.return_value = /dir/libdir confDir_mock.return_value = /dir/confdir +os_env_get_mock.return_value = env_val slider.main() self.assertTrue(java_mock.called) -java_mock.assert_called_with( - 'org.apache.slider.Slider', - ['list'], - '/dir/libdir/*:/dir/confdir::', - ['-Dslider.confdir=/dir/confdir', '-Dslider.libdir=/dir/libdir', '-Djava.net.preferIPv4Stack=true', '-Djava.awt.headless=true', '-Xmx256m']) +if IS_WINDOWS: + java_mock.assert_called_with( +'org.apache.slider.Slider', +['list'], +'/dir/libdir\\*;/dir/confdir;env_val;env_val', +['-Dslider.confdir=/dir/confdir', '-Dslider.libdir=/dir/libdir', 'env_val']) +else: + java_mock.assert_called_with( +'org.apache.slider.Slider', +['list'], +'/dir/libdir/*:/dir/confdir:env_val:env_val', +['-Dslider.confdir=/dir/confdir', '-Dslider.libdir=/dir/libdir', 'env_val']) pass
[1/3] incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 d6299b746 - a036ec112
SLIDER-630. slider-agent unit tests fail on debian and suse and windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/16a48dff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/16a48dff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/16a48dff
Branch: refs/heads/releases/slider-0.60
Commit: 16a48dff95ce044ba98d2ffdf7b2081a45f45237
Parents: d6299b7
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sun Nov 9 14:26:54 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sun Nov 9 15:07:00 2014 -0800
--
slider-agent/src/test/python/python-wrap| 13 +
.../resource_management/TestExecuteResource.py | 2 +-
slider-agent/src/test/python/unitTests.py | 2 +-
slider-assembly/pom.xml | 28 ++--
slider-assembly/src/test/python/unitTests.py| 2 +-
5 files changed, 37 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/16a48dff/slider-agent/src/test/python/python-wrap
--
diff --git a/slider-agent/src/test/python/python-wrap
b/slider-agent/src/test/python/python-wrap
index ea57721..88a8c55 100755
--- a/slider-agent/src/test/python/python-wrap
+++ b/slider-agent/src/test/python/python-wrap
@@ -17,23 +17,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-export PYTHONPATH=/usr/lib/python2.6/site-packages:$PYTHONPATH
-
# reset settings
unset PYTHON
+if [ -a /usr/bin/python2.6 ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python2.6
+fi
+
# checking for preferable python versions
if [ -a /usr/bin/python2.7 ] [ -z $PYTHON ]; then
PYTHON=/usr/bin/python2.7
fi
-if [ -a /usr/bin/python2.6 ] [ -z $PYTHON ]; then
- PYTHON=/usr/bin/python2.6
+# if no preferable python versions found, try to use system one
+if [ -a /usr/bin/python ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python
fi
# if no preferable python versions found, try to use system one
if [[ -z $PYTHON ]]; then
- PYTHON=/usr/bin/python
+ PYTHON=python
fi
# execute script
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/16a48dff/slider-agent/src/test/python/resource_management/TestExecuteResource.py
--
diff --git
a/slider-agent/src/test/python/resource_management/TestExecuteResource.py
b/slider-agent/src/test/python/resource_management/TestExecuteResource.py
index f7f6371..0673b66 100644
--- a/slider-agent/src/test/python/resource_management/TestExecuteResource.py
+++ b/slider-agent/src/test/python/resource_management/TestExecuteResource.py
@@ -77,7 +77,7 @@ class TestExecuteResource(TestCase):
poll_after = 5)
self.assertTrue(False, Should fail as process does not run for 5
seconds)
except Fail as e:
-self.assertTrue(returned 1 in e.message)
+self.assertTrue(returned 1 in str(e))
pass
self.assertTrue(popen_mock.called, 'subprocess.Popen should have been
called!')
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/16a48dff/slider-agent/src/test/python/unitTests.py
--
diff --git a/slider-agent/src/test/python/unitTests.py
b/slider-agent/src/test/python/unitTests.py
index 6aa0167..b65c075 100644
--- a/slider-agent/src/test/python/unitTests.py
+++ b/slider-agent/src/test/python/unitTests.py
@@ -32,7 +32,7 @@ ignoredDirs = [mock]
class TestAgent(unittest.TestSuite):
def run(self, result, debug=False):
run = unittest.TestSuite.run
-run(self, result, debug)
+run(self, result)
return result
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/16a48dff/slider-assembly/pom.xml
--
diff --git a/slider-assembly/pom.xml b/slider-assembly/pom.xml
index fa6f1ad..b6f20c0 100644
--- a/slider-assembly/pom.xml
+++ b/slider-assembly/pom.xml
@@ -38,6 +38,9 @@
src.confdirsrc/conf-hdp/src.confdir
src.libdir${project.build.directory}/lib/src.libdir
src.agent.ini.dir${project.build.directory}/../../slider-agent/conf/src.agent.ini.dir
+python.verpython gt;= 2.6/python.ver
+
executable.python${project.basedir}/../slider-agent/src/test/python/python-wrap/executable.python
+
python.path.l${project.basedir}/src/main/scripts:${project.basedir}/../slider-agent/src/test/python/mock:${project.basedir}/src/test/python/scripts/python.path.l
skipTestsfalse/skipTests
/properties
@@ -72,13
[2/3] incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse and windows
SLIDER-630. slider-agent unit tests fail on debian and suse and windows Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/131221e3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/131221e3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/131221e3 Branch: refs/heads/releases/slider-0.60 Commit: 131221e38a9bca00f63383fb566203ebfcc0a985 Parents: 16a48df Author: Sumit Mohanty smoha...@hortonworks.com Authored: Sun Nov 9 14:34:38 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Sun Nov 9 15:07:12 2014 -0800 -- slider-agent/src/test/python/agent/TestPythonExecutor.py | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/131221e3/slider-agent/src/test/python/agent/TestPythonExecutor.py -- diff --git a/slider-agent/src/test/python/agent/TestPythonExecutor.py b/slider-agent/src/test/python/agent/TestPythonExecutor.py index 9a46bd5..0a43639 100644 --- a/slider-agent/src/test/python/agent/TestPythonExecutor.py +++ b/slider-agent/src/test/python/agent/TestPythonExecutor.py @@ -33,6 +33,7 @@ from PythonExecutor import PythonExecutor from AgentConfig import AgentConfig from mock.mock import MagicMock, patch from AgentToggleLogger import AgentToggleLogger +import os class TestPythonExecutor(TestCase): def setUp(self):
incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse
Repository: incubator-slider
Updated Branches:
refs/heads/develop 1d11d626c - e74e7fe08
SLIDER-630. slider-agent unit tests fail on debian and suse
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/e74e7fe0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/e74e7fe0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/e74e7fe0
Branch: refs/heads/develop
Commit: e74e7fe08ba12d73e277303863a8f0eb728cef02
Parents: 1d11d62
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sat Nov 8 13:53:40 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sat Nov 8 13:53:40 2014 -0800
--
slider-agent/pom.xml| 2 +-
slider-agent/src/test/python/python-wrap| 40 ++
slider-agent/src/test/python/python-wrap.sh | 42
3 files changed, 41 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e74e7fe0/slider-agent/pom.xml
--
diff --git a/slider-agent/pom.xml b/slider-agent/pom.xml
index 14c7be4..bcfc209 100644
--- a/slider-agent/pom.xml
+++ b/slider-agent/pom.xml
@@ -63,7 +63,7 @@
executions
execution
configuration
- executablepython/executable
+
executable${project.basedir}/../slider-agent/src/test/python/python-wrap/executable
workingDirectorysrc/test/python/workingDirectory
arguments
argumentunitTests.py/argument
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e74e7fe0/slider-agent/src/test/python/python-wrap
--
diff --git a/slider-agent/src/test/python/python-wrap
b/slider-agent/src/test/python/python-wrap
new file mode 100755
index 000..ea57721
--- /dev/null
+++ b/slider-agent/src/test/python/python-wrap
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Copyright 2011 The Apache Software Foundation
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export PYTHONPATH=/usr/lib/python2.6/site-packages:$PYTHONPATH
+
+# reset settings
+unset PYTHON
+
+# checking for preferable python versions
+if [ -a /usr/bin/python2.7 ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python2.7
+fi
+
+if [ -a /usr/bin/python2.6 ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python2.6
+fi
+
+# if no preferable python versions found, try to use system one
+if [[ -z $PYTHON ]]; then
+ PYTHON=/usr/bin/python
+fi
+
+# execute script
+$PYTHON $@
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/e74e7fe0/slider-agent/src/test/python/python-wrap.sh
--
diff --git a/slider-agent/src/test/python/python-wrap.sh
b/slider-agent/src/test/python/python-wrap.sh
deleted file mode 100755
index 8f30d67..000
--- a/slider-agent/src/test/python/python-wrap.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#!/usr/bin/env bash
-# Copyright 2011 The Apache Software Foundation
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# License); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an AS IS BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-export PYTHONPATH=/usr/lib/python2.7/site-packages/common_functions:$PYTHONPATH
-
-# reset settings
-unset PYTHON
-
-# checking for preferable python versions
-if [ -a
incubator-slider git commit: SLIDER-630. slider-agent unit tests fail on debian and suse
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 ad4697575 - bf5beef60
SLIDER-630. slider-agent unit tests fail on debian and suse
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/bf5beef6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/bf5beef6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/bf5beef6
Branch: refs/heads/releases/slider-0.60
Commit: bf5beef60c2dcff7833b47ecb7794a7501467656
Parents: ad46975
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sat Nov 8 13:53:40 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sat Nov 8 14:55:09 2014 -0800
--
slider-agent/pom.xml| 2 +-
slider-agent/src/test/python/python-wrap| 40 ++
slider-agent/src/test/python/python-wrap.sh | 42
3 files changed, 41 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bf5beef6/slider-agent/pom.xml
--
diff --git a/slider-agent/pom.xml b/slider-agent/pom.xml
index 95af5ba..4e5b6bd 100644
--- a/slider-agent/pom.xml
+++ b/slider-agent/pom.xml
@@ -63,7 +63,7 @@
executions
execution
configuration
- executablepython/executable
+
executable${project.basedir}/../slider-agent/src/test/python/python-wrap/executable
workingDirectorysrc/test/python/workingDirectory
arguments
argumentunitTests.py/argument
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bf5beef6/slider-agent/src/test/python/python-wrap
--
diff --git a/slider-agent/src/test/python/python-wrap
b/slider-agent/src/test/python/python-wrap
new file mode 100755
index 000..ea57721
--- /dev/null
+++ b/slider-agent/src/test/python/python-wrap
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Copyright 2011 The Apache Software Foundation
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export PYTHONPATH=/usr/lib/python2.6/site-packages:$PYTHONPATH
+
+# reset settings
+unset PYTHON
+
+# checking for preferable python versions
+if [ -a /usr/bin/python2.7 ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python2.7
+fi
+
+if [ -a /usr/bin/python2.6 ] [ -z $PYTHON ]; then
+ PYTHON=/usr/bin/python2.6
+fi
+
+# if no preferable python versions found, try to use system one
+if [[ -z $PYTHON ]]; then
+ PYTHON=/usr/bin/python
+fi
+
+# execute script
+$PYTHON $@
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/bf5beef6/slider-agent/src/test/python/python-wrap.sh
--
diff --git a/slider-agent/src/test/python/python-wrap.sh
b/slider-agent/src/test/python/python-wrap.sh
deleted file mode 100755
index 8f30d67..000
--- a/slider-agent/src/test/python/python-wrap.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#!/usr/bin/env bash
-# Copyright 2011 The Apache Software Foundation
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# License); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an AS IS BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-export PYTHONPATH=/usr/lib/python2.7/site-packages/common_functions:$PYTHONPATH
-
-# reset settings
-unset PYTHON
-
-# checking for preferable
incubator-slider git commit: SLIDER-620. Execute command should allow polling for daemons that may stop within seconds
Repository: incubator-slider
Updated Branches:
refs/heads/develop 3585e0b5a - d77734ad8
SLIDER-620. Execute command should allow polling for daemons that may stop
within seconds
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/d77734ad
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/d77734ad
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/d77734ad
Branch: refs/heads/develop
Commit: d77734ad8c2a8a4c20a13c51e11224beebfeafb4
Parents: 3585e0b
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Nov 6 18:29:41 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 6 18:29:41 2014 -0800
--
.../hbase-win/package/scripts/params.py | 16 +-
app-packages/memcached-win/metainfo.xml | 18 +++
.../memcached-win/package/scripts/memcached.py | 6 ++--
.../memcached-win/package/scripts/params.py | 2 +-
app-packages/memcached/README.txt | 11 ++-
.../memcached/package/scripts/memcached.py | 3 +-
.../python/resource_management/core/logger.py | 20 ++--
.../core/providers/system.py| 2 +-
.../core/providers/windows/system.py| 33 +++-
.../core/resources/system.py| 1 +
.../python/resource_management/core/shell.py| 29 -
.../libraries/functions/check_process_status.py | 2 +-
.../resource_management/TestExecuteResource.py | 25 +++
13 files changed, 128 insertions(+), 40 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d77734ad/app-packages/hbase-win/package/scripts/params.py
--
diff --git a/app-packages/hbase-win/package/scripts/params.py
b/app-packages/hbase-win/package/scripts/params.py
index 90d274c..5a54e25 100644
--- a/app-packages/hbase-win/package/scripts/params.py
+++ b/app-packages/hbase-win/package/scripts/params.py
@@ -44,11 +44,11 @@ hbase_hdfs_root_dir =
config['configurations']['hbase-site']['hbase.rootdir']
Read various ports
-rest_port = default(configurations/global/hbase_rest_port, 1700)
-thrift_port = default(configurations/global/hbase_thrift_port, 9090)
-thrift2_port = default(configurations/global/hbase_thrift2_port, 9091)
-thrift_info_port = default(configurations/global/hbase_info_thrift_port,
9095)
-thrift2_info_port = default(configurations/global/hbase_info_thrift2_port,
9096)
+rest_port = default(/configurations/global/hbase_rest_port, 1700)
+thrift_port = default(/configurations/global/hbase_thrift_port, 9090)
+thrift2_port = default(/configurations/global/hbase_thrift2_port, 9091)
+thrift_info_port = default(/configurations/global/hbase_info_thrift_port,
9095)
+thrift2_info_port = default(/configurations/global/hbase_info_thrift2_port,
9096)
Compute or read various heap sizes
@@ -59,9 +59,9 @@ regionserver_xmn_max =
config['configurations']['hbase-env']['hbase_regionserver
regionserver_xmn_percent =
config['configurations']['hbase-env']['hbase_regionserver_xmn_ratio']
regionserver_xmn_size = calc_xmn_from_xms(regionserver_heapsize,
regionserver_xmn_percent, regionserver_xmn_max)
-restserver_heapsize =
default(configurations/hbase-env/hbase_restserver_heapsize, 512m)
-thriftserver_heapsize =
default(configurations/hbase-env/hbase_thriftserver_heapsize, 512m)
-thrift2server_heapsize =
default(configurations/hbase-env/hbase_thrift2server_heapsize, 512m)
+restserver_heapsize =
default(/configurations/hbase-env/hbase_restserver_heapsize, 512m)
+thriftserver_heapsize =
default(/configurations/hbase-env/hbase_thriftserver_heapsize, 512m)
+thrift2server_heapsize =
default(/configurations/hbase-env/hbase_thrift2server_heapsize, 512m)
hbase_env_sh_template = config['configurations']['hbase-env']['content']
java_library_path = config['configurations']['global']['java_library_path']
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/d77734ad/app-packages/memcached-win/metainfo.xml
--
diff --git a/app-packages/memcached-win/metainfo.xml
b/app-packages/memcached-win/metainfo.xml
index c7e5881..093001b 100644
--- a/app-packages/memcached-win/metainfo.xml
+++ b/app-packages/memcached-win/metainfo.xml
@@ -23,17 +23,23 @@
commentMemcache is a network accessible key/value storage system, often
used as a distributed cache./comment
version1.0.0/version
exportedConfigsNone/exportedConfigs
+exportGroups
+ exportGroup
+nameServers/name
+exports
+ export
+namehost_port/name
+value${MEMCACHED_HOST}:${site.global.listen_port}/value
+ /export
+
incubator-slider git commit: SLIDER-620. Execute command should allow polling for daemons that may stop within seconds
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 b0a773cf3 - 9641bd0c6
SLIDER-620. Execute command should allow polling for daemons that may stop
within seconds
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/9641bd0c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/9641bd0c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/9641bd0c
Branch: refs/heads/releases/slider-0.60
Commit: 9641bd0c6860d6e06b84adf6123c021249b2ed82
Parents: b0a773c
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Thu Nov 6 18:29:41 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Thu Nov 6 18:31:55 2014 -0800
--
.../hbase-win/package/scripts/params.py | 16 +-
app-packages/memcached-win/metainfo.xml | 18 +++
.../memcached-win/package/scripts/memcached.py | 6 ++--
.../memcached-win/package/scripts/params.py | 2 +-
app-packages/memcached/README.txt | 11 ++-
.../memcached/package/scripts/memcached.py | 3 +-
.../python/resource_management/core/logger.py | 20 ++--
.../core/providers/system.py| 2 +-
.../core/providers/windows/system.py| 33 +++-
.../core/resources/system.py| 1 +
.../python/resource_management/core/shell.py| 29 -
.../libraries/functions/check_process_status.py | 2 +-
.../resource_management/TestExecuteResource.py | 25 +++
13 files changed, 128 insertions(+), 40 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9641bd0c/app-packages/hbase-win/package/scripts/params.py
--
diff --git a/app-packages/hbase-win/package/scripts/params.py
b/app-packages/hbase-win/package/scripts/params.py
index 90d274c..5a54e25 100644
--- a/app-packages/hbase-win/package/scripts/params.py
+++ b/app-packages/hbase-win/package/scripts/params.py
@@ -44,11 +44,11 @@ hbase_hdfs_root_dir =
config['configurations']['hbase-site']['hbase.rootdir']
Read various ports
-rest_port = default(configurations/global/hbase_rest_port, 1700)
-thrift_port = default(configurations/global/hbase_thrift_port, 9090)
-thrift2_port = default(configurations/global/hbase_thrift2_port, 9091)
-thrift_info_port = default(configurations/global/hbase_info_thrift_port,
9095)
-thrift2_info_port = default(configurations/global/hbase_info_thrift2_port,
9096)
+rest_port = default(/configurations/global/hbase_rest_port, 1700)
+thrift_port = default(/configurations/global/hbase_thrift_port, 9090)
+thrift2_port = default(/configurations/global/hbase_thrift2_port, 9091)
+thrift_info_port = default(/configurations/global/hbase_info_thrift_port,
9095)
+thrift2_info_port = default(/configurations/global/hbase_info_thrift2_port,
9096)
Compute or read various heap sizes
@@ -59,9 +59,9 @@ regionserver_xmn_max =
config['configurations']['hbase-env']['hbase_regionserver
regionserver_xmn_percent =
config['configurations']['hbase-env']['hbase_regionserver_xmn_ratio']
regionserver_xmn_size = calc_xmn_from_xms(regionserver_heapsize,
regionserver_xmn_percent, regionserver_xmn_max)
-restserver_heapsize =
default(configurations/hbase-env/hbase_restserver_heapsize, 512m)
-thriftserver_heapsize =
default(configurations/hbase-env/hbase_thriftserver_heapsize, 512m)
-thrift2server_heapsize =
default(configurations/hbase-env/hbase_thrift2server_heapsize, 512m)
+restserver_heapsize =
default(/configurations/hbase-env/hbase_restserver_heapsize, 512m)
+thriftserver_heapsize =
default(/configurations/hbase-env/hbase_thriftserver_heapsize, 512m)
+thrift2server_heapsize =
default(/configurations/hbase-env/hbase_thrift2server_heapsize, 512m)
hbase_env_sh_template = config['configurations']['hbase-env']['content']
java_library_path = config['configurations']['global']['java_library_path']
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9641bd0c/app-packages/memcached-win/metainfo.xml
--
diff --git a/app-packages/memcached-win/metainfo.xml
b/app-packages/memcached-win/metainfo.xml
index c7e5881..093001b 100644
--- a/app-packages/memcached-win/metainfo.xml
+++ b/app-packages/memcached-win/metainfo.xml
@@ -23,17 +23,23 @@
commentMemcache is a network accessible key/value storage system, often
used as a distributed cache./comment
version1.0.0/version
exportedConfigsNone/exportedConfigs
+exportGroups
+ exportGroup
+nameServers/name
+exports
+ export
+namehost_port/name
+value${MEMCACHED_HOST}:${site.global.listen_port}/value
git commit: Modify few default values for Storm Windows package
Repository: incubator-slider
Updated Branches:
refs/heads/develop 4e4652e4b - 9f5dbb467
Modify few default values for Storm Windows package
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/9f5dbb46
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/9f5dbb46
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/9f5dbb46
Branch: refs/heads/develop
Commit: 9f5dbb4672dae2d1727f851dd15d3c9facdc0560
Parents: 4e4652e
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Nov 5 16:36:06 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Nov 5 16:36:06 2014 -0800
--
app-packages/storm-win/configuration/storm-site.xml | 4 ++--
app-packages/storm/appConfig-secured-default.json | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9f5dbb46/app-packages/storm-win/configuration/storm-site.xml
--
diff --git a/app-packages/storm-win/configuration/storm-site.xml
b/app-packages/storm-win/configuration/storm-site.xml
index 86db125..c09e29b 100644
--- a/app-packages/storm-win/configuration/storm-site.xml
+++ b/app-packages/storm-win/configuration/storm-site.xml
@@ -118,7 +118,7 @@
/property
property
namenimbus.childopts/name
-value-Xmx1024m
-Djava.security.auth.login.config=/etc/storm/conf/storm_jaas.conf
-javaagent:/usr/lib/storm/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/lib/storm/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM/value
+value-Xmx1024m/value
descriptionThis parameter is used by the storm-deploy project to
configure the jvm options for the nimbus daemon./description
/property
property
@@ -292,7 +292,7 @@
/property
property
nameworker.childopts/name
-value-Xmx768m
-javaagent:/usr/lib/storm/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/lib/storm/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM/value
+value-Xmx768m/value
descriptionThe jvm opts provided to workers launched by this supervisor.
All \%ID%\ substrings are replaced with an identifier for this
worker./description
/property
property
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/9f5dbb46/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index bd7ac59..ec42acd 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -50,7 +50,7 @@
site.storm-site.ui.filter.params: {'type': 'kerberos',
'kerberos.principal': 'HTTP/_HOST', 'kerberos.keytab':
'/etc/security/keytabs/spnego.service.keytab', 'kerberos.name.rules':
'RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT'},
site.storm-env.kerberos_domain: EXAMPLE.COM,
-site.storm-env.storm_client_principal_name:
PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE/_HOST,
+site.storm-env.storm_client_principal_name:
PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE,
site.storm-env.storm_server_principal_name:
PLACE_JAAS_STORMSERVER_PRINCIPAL_HERE/_HOST,
site.storm-env.storm_client_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.client.keytab,
site.storm-env.storm_server_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.nimbus.keytab
git commit: Modify few default values for Storm Windows package
Repository: incubator-slider
Updated Branches:
refs/heads/releases/slider-0.60 5034161db - 25c09e088
Modify few default values for Storm Windows package
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/25c09e08
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/25c09e08
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/25c09e08
Branch: refs/heads/releases/slider-0.60
Commit: 25c09e0887aa6ce531d8de52fa6f5bd89b39c370
Parents: 5034161
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Wed Nov 5 16:36:06 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Wed Nov 5 16:37:17 2014 -0800
--
app-packages/storm-win/configuration/storm-site.xml | 4 ++--
app-packages/storm/appConfig-secured-default.json | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/25c09e08/app-packages/storm-win/configuration/storm-site.xml
--
diff --git a/app-packages/storm-win/configuration/storm-site.xml
b/app-packages/storm-win/configuration/storm-site.xml
index 86db125..c09e29b 100644
--- a/app-packages/storm-win/configuration/storm-site.xml
+++ b/app-packages/storm-win/configuration/storm-site.xml
@@ -118,7 +118,7 @@
/property
property
namenimbus.childopts/name
-value-Xmx1024m
-Djava.security.auth.login.config=/etc/storm/conf/storm_jaas.conf
-javaagent:/usr/lib/storm/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/lib/storm/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM/value
+value-Xmx1024m/value
descriptionThis parameter is used by the storm-deploy project to
configure the jvm options for the nimbus daemon./description
/property
property
@@ -292,7 +292,7 @@
/property
property
nameworker.childopts/name
-value-Xmx768m
-javaagent:/usr/lib/storm/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/lib/storm/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM/value
+value-Xmx768m/value
descriptionThe jvm opts provided to workers launched by this supervisor.
All \%ID%\ substrings are replaced with an identifier for this
worker./description
/property
property
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/25c09e08/app-packages/storm/appConfig-secured-default.json
--
diff --git a/app-packages/storm/appConfig-secured-default.json
b/app-packages/storm/appConfig-secured-default.json
index bd7ac59..ec42acd 100644
--- a/app-packages/storm/appConfig-secured-default.json
+++ b/app-packages/storm/appConfig-secured-default.json
@@ -50,7 +50,7 @@
site.storm-site.ui.filter.params: {'type': 'kerberos',
'kerberos.principal': 'HTTP/_HOST', 'kerberos.keytab':
'/etc/security/keytabs/spnego.service.keytab', 'kerberos.name.rules':
'RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT'},
site.storm-env.kerberos_domain: EXAMPLE.COM,
-site.storm-env.storm_client_principal_name:
PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE/_HOST,
+site.storm-env.storm_client_principal_name:
PLACE_JAAS_STORMCLIENT_PRINCIPAL_HERE,
site.storm-env.storm_server_principal_name:
PLACE_JAAS_STORMSERVER_PRINCIPAL_HERE/_HOST,
site.storm-env.storm_client_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.client.keytab,
site.storm-env.storm_server_keytab:
${AGENT_WORK_ROOT}/keytabs/${USER_NAME}.STORM.nimbus.keytab
git commit: Nimbus should publish config for Storm
Repository: incubator-slider Updated Branches: refs/heads/develop a29d825e6 - fe88bbd18 Nimbus should publish config for Storm Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/fe88bbd1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/fe88bbd1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/fe88bbd1 Branch: refs/heads/develop Commit: fe88bbd1895398fb71de91ffa474629ac6f8e326 Parents: a29d825 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Wed Nov 5 17:59:43 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Wed Nov 5 17:59:43 2014 -0800 -- app-packages/storm-win/metainfo.xml | 1 + app-packages/storm/metainfo.xml | 1 + 2 files changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fe88bbd1/app-packages/storm-win/metainfo.xml -- diff --git a/app-packages/storm-win/metainfo.xml b/app-packages/storm-win/metainfo.xml index ec68ef6..1e9e7ab 100644 --- a/app-packages/storm-win/metainfo.xml +++ b/app-packages/storm-win/metainfo.xml @@ -69,6 +69,7 @@ component nameNIMBUS/name categoryMASTER/category +publishConfigtrue/publishConfig autoStartOnFailuretrue/autoStartOnFailure appExportsQuickLinks-nimbus.host_port/appExports commandScript http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fe88bbd1/app-packages/storm/metainfo.xml -- diff --git a/app-packages/storm/metainfo.xml b/app-packages/storm/metainfo.xml index e07ae92..28b0e9b 100644 --- a/app-packages/storm/metainfo.xml +++ b/app-packages/storm/metainfo.xml @@ -76,6 +76,7 @@ component nameNIMBUS/name categoryMASTER/category +publishConfigtrue/publishConfig autoStartOnFailuretrue/autoStartOnFailure appExportsQuickLinks-nimbus.host_port,QuickLinks-org.apache.slider.metrics.ui,QuickLinks-org.apache.slider.metrics/appExports maxInstanceCount1/maxInstanceCount
git commit: Nimbus should publish config for Storm
Repository: incubator-slider Updated Branches: refs/heads/releases/slider-0.60 25c09e088 - ad9fa1771 Nimbus should publish config for Storm Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/ad9fa177 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/ad9fa177 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/ad9fa177 Branch: refs/heads/releases/slider-0.60 Commit: ad9fa1771122a04a4c760f5c878258b3135d6ef0 Parents: 25c09e0 Author: Sumit Mohanty smoha...@hortonworks.com Authored: Wed Nov 5 17:59:43 2014 -0800 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Wed Nov 5 18:00:20 2014 -0800 -- app-packages/storm-win/metainfo.xml | 1 + app-packages/storm/metainfo.xml | 1 + 2 files changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/ad9fa177/app-packages/storm-win/metainfo.xml -- diff --git a/app-packages/storm-win/metainfo.xml b/app-packages/storm-win/metainfo.xml index ec68ef6..1e9e7ab 100644 --- a/app-packages/storm-win/metainfo.xml +++ b/app-packages/storm-win/metainfo.xml @@ -69,6 +69,7 @@ component nameNIMBUS/name categoryMASTER/category +publishConfigtrue/publishConfig autoStartOnFailuretrue/autoStartOnFailure appExportsQuickLinks-nimbus.host_port/appExports commandScript http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/ad9fa177/app-packages/storm/metainfo.xml -- diff --git a/app-packages/storm/metainfo.xml b/app-packages/storm/metainfo.xml index e07ae92..28b0e9b 100644 --- a/app-packages/storm/metainfo.xml +++ b/app-packages/storm/metainfo.xml @@ -76,6 +76,7 @@ component nameNIMBUS/name categoryMASTER/category +publishConfigtrue/publishConfig autoStartOnFailuretrue/autoStartOnFailure appExportsQuickLinks-nimbus.host_port,QuickLinks-org.apache.slider.metrics.ui,QuickLinks-org.apache.slider.metrics/appExports maxInstanceCount1/maxInstanceCount
git commit: SLIDER-525. Storm app package for windows does not report component status correctly
Repository: incubator-slider
Updated Branches:
refs/heads/develop 497399904 - 6c015b089
SLIDER-525. Storm app package for windows does not report component status
correctly
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6c015b08
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6c015b08
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6c015b08
Branch: refs/heads/develop
Commit: 6c015b089fdfca3e6b48e4e49a8af106196fc9e6
Parents: 4973999
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Tue Nov 4 11:46:28 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Tue Nov 4 11:46:28 2014 -0800
--
app-packages/storm-win/appConfig-default.json | 2 +-
.../storm-win/configuration/storm-site.xml | 4 ++--
.../storm-win/package/scripts/params.py | 15 ---
.../storm-win/package/scripts/service.py| 2 ++
app-packages/storm-win/package/scripts/storm.py | 8
.../storm-win/package/templates/config.yaml.j2 | 20 +++-
6 files changed, 24 insertions(+), 27 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6c015b08/app-packages/storm-win/appConfig-default.json
--
diff --git a/app-packages/storm-win/appConfig-default.json
b/app-packages/storm-win/appConfig-default.json
index 229bc1d..a77f00d 100644
--- a/app-packages/storm-win/appConfig-default.json
+++ b/app-packages/storm-win/appConfig-default.json
@@ -10,7 +10,6 @@
site.global.app_user: hadoop,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/storm-${pkg.version},
site.global.user_group: hadoop,
-site.global.security_enabled: false,
site.storm-site.storm.log.dir : ${AGENT_LOG_ROOT},
site.storm-site.storm.zookeeper.servers: ['${ZK_HOST}'],
@@ -20,6 +19,7 @@
site.storm-site.storm.zookeeper.port: 2181,
site.storm-site.nimbus.childopts: -Xmx1024m,
site.storm-site.worker.childopts: -Xmx768m,
+site.storm-site.ui.childopts: -Xmx768m,
site.storm-site.dev.zookeeper.path:
${AGENT_WORK_ROOT}/app/tmp/dev-storm-zookeeper,
site.storm-site.drpc.invocations.port: 0,
site.storm-site.storm.zookeeper.root: ${DEFAULT_ZK_PATH},
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6c015b08/app-packages/storm-win/configuration/storm-site.xml
--
diff --git a/app-packages/storm-win/configuration/storm-site.xml
b/app-packages/storm-win/configuration/storm-site.xml
index b3cce6a..86db125 100644
--- a/app-packages/storm-win/configuration/storm-site.xml
+++ b/app-packages/storm-win/configuration/storm-site.xml
@@ -188,7 +188,7 @@
/property
property
nameui.childopts/name
-value-Xmx768m
-Djava.security.auth.login.config=/etc/storm/conf/storm_jaas.conf/value
+value-Xmx768m/value
descriptionChildopts for Storm UI Java process./description
/property
property
@@ -264,7 +264,7 @@
/property
property
namesupervisor.childopts/name
-value-Xmx256m
-Djava.security.auth.login.config=/etc/storm/conf/storm_jaas.conf
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=56431
-javaagent:/usr/lib/storm/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/lib/storm/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM/value
+value-Xmx256m/value
descriptionThis parameter is used by the storm-deploy project to
configure the jvm options for the supervisor daemon./description
/property
property
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6c015b08/app-packages/storm-win/package/scripts/params.py
--
diff --git a/app-packages/storm-win/package/scripts/params.py
b/app-packages/storm-win/package/scripts/params.py
index 3a458ea..21e5c65 100644
--- a/app-packages/storm-win/package/scripts/params.py
+++ b/app-packages/storm-win/package/scripts/params.py
@@ -37,18 +37,3 @@ nimbus_port =
config['configurations']['storm-site']['nimbus.thrift.port']
rest_api_conf_file = format({conf_dir}/config.yaml)
rest_lib_dir = format({app_root}/external/storm-rest)
storm_bin = format({app_root}/bin/storm.cmd)
-
-security_enabled = config['configurations']['global']['security_enabled']
-
-if security_enabled:
- _hostname_lowercase = config['hostname'].lower()
- _kerberos_domain = config['configurations']['storm-env']['kerberos_domain']
- _storm_client_principal_name =
git commit: Storm app package changes to fix the jmx property names and default yarn.memory values
Repository: incubator-slider
Updated Branches:
refs/heads/develop 6218b2d0e - 0b61c4575
Storm app package changes to fix the jmx property names and default yarn.memory
values
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0b61c457
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0b61c457
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0b61c457
Branch: refs/heads/develop
Commit: 0b61c45754d1b0fb53628bf3451a28a28d3a31a0
Parents: 6218b2d
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Mon Nov 3 15:09:08 2014 -0800
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Mon Nov 3 15:09:08 2014 -0800
--
app-packages/hbase-win/appConfig-default.json | 2 --
app-packages/storm/jmx_metrics.json | 8
app-packages/storm/resources-default.json | 13 +
3 files changed, 13 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0b61c457/app-packages/hbase-win/appConfig-default.json
--
diff --git a/app-packages/hbase-win/appConfig-default.json
b/app-packages/hbase-win/appConfig-default.json
index 80d20af..04cb9a9 100644
--- a/app-packages/hbase-win/appConfig-default.json
+++ b/app-packages/hbase-win/appConfig-default.json
@@ -10,9 +10,7 @@
site.global.app_user: hadoop,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/hbase-${pkg.version},
site.global.hbase_instance_name: instancename,
-site.global.hbase_root_password: secret,
site.global.user_group: hadoop,
-site.global.monitor_protocol: http,
site.global.hbase_additional_cp: c:\\java\\lib\\tools.jar;,
site.global.java_library_path: c:\\hdp\\hadoop\\bin,
site.global.hbase_rest_port: 17000,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0b61c457/app-packages/storm/jmx_metrics.json
--
diff --git a/app-packages/storm/jmx_metrics.json
b/app-packages/storm/jmx_metrics.json
index f7d4e60..b0816b1 100644
--- a/app-packages/storm/jmx_metrics.json
+++ b/app-packages/storm/jmx_metrics.json
@@ -2,17 +2,17 @@
Component: {
NIMBUS: {
FreeSlots: {
-metric: $['slots.free'],
+metric: $['slotsFree'],
pointInTime: true,
temporal: false
},
Tasks: {
-metric: $['tasks.total'],
+metric: $['tasksTotal'],
pointInTime: true,
temporal: false
},
Executors: {
-metric: $['executors.total'],
+metric: $['executorsTotal'],
pointInTime: true,
temporal: false
},
@@ -22,7 +22,7 @@
temporal: false
},
NimbusUptime: {
-metric: $['nimbus.uptime'],
+metric: $['nimbusUptime'],
pointInTime: true,
temporal: false
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0b61c457/app-packages/storm/resources-default.json
--
diff --git a/app-packages/storm/resources-default.json
b/app-packages/storm/resources-default.json
index a36f005..65e6579 100644
--- a/app-packages/storm/resources-default.json
+++ b/app-packages/storm/resources-default.json
@@ -8,22 +8,27 @@
},
components: {
slider-appmaster: {
+ yarn.memory: 512
},
NIMBUS: {
yarn.role.priority: 1,
- yarn.component.instances: 1
+ yarn.component.instances: 1,
+ yarn.memory: 2048
},
STORM_UI_SERVER: {
yarn.role.priority: 2,
- yarn.component.instances: 1
+ yarn.component.instances: 1,
+ yarn.memory: 1278
},
DRPC_SERVER: {
yarn.role.priority: 3,
- yarn.component.instances: 1
+ yarn.component.instances: 1,
+ yarn.memory: 1278
},
SUPERVISOR: {
yarn.role.priority: 4,
- yarn.component.instances: 1
+ yarn.component.instances: 1,
+ yarn.memory: 3072
}
}
}
git commit: SLIDER-575. Slider fun-test stabilization (HBase fun tests)
Repository: incubator-slider
Updated Branches:
refs/heads/develop b072b9fd6 - b01927162
SLIDER-575. Slider fun-test stabilization (HBase fun tests)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/b0192716
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/b0192716
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/b0192716
Branch: refs/heads/develop
Commit: b019271621530cd2dde2f0a44a557d8ee8e52c42
Parents: b072b9f
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sat Nov 1 14:05:27 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sat Nov 1 14:05:27 2014 -0700
--
app-packages/hbase/package/scripts/hbase_service.py | 4 ++--
app-packages/hbase/resources-default.json| 1 +
.../main/java/org/apache/slider/common/tools/CoreFileSystem.java | 4
3 files changed, 7 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b0192716/app-packages/hbase/package/scripts/hbase_service.py
--
diff --git a/app-packages/hbase/package/scripts/hbase_service.py
b/app-packages/hbase/package/scripts/hbase_service.py
index 96add84..db663b8 100644
--- a/app-packages/hbase/package/scripts/hbase_service.py
+++ b/app-packages/hbase/package/scripts/hbase_service.py
@@ -34,7 +34,7 @@ def hbase_service(
no_op_test = None
if action == 'start':
- daemon_cmd = format({cmd} start {role})
+ daemon_cmd = format(env HBASE_IDENT_STRING={hbase_user} {cmd} start
{role})
if name == 'rest':
daemon_cmd = format({daemon_cmd} -p {rest_port})
elif name == 'thrift':
@@ -43,7 +43,7 @@ def hbase_service(
daemon_cmd = format({daemon_cmd} -p {thrift2_port})
no_op_test = format(ls {pid_file} /dev/null 21 ps `cat
{pid_file}` /dev/null 21)
elif action == 'stop':
- daemon_cmd = format({cmd} stop {role} rm -f {pid_file})
+ daemon_cmd = format(env HBASE_IDENT_STRING={hbase_user} {cmd} stop
{role} rm -f {pid_file})
if daemon_cmd is not None:
Execute ( daemon_cmd,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b0192716/app-packages/hbase/resources-default.json
--
diff --git a/app-packages/hbase/resources-default.json
b/app-packages/hbase/resources-default.json
index ff58989..b1da1f7 100644
--- a/app-packages/hbase/resources-default.json
+++ b/app-packages/hbase/resources-default.json
@@ -13,6 +13,7 @@
yarn.memory: 1500
},
slider-appmaster: {
+ yarn.memory: 1024
},
HBASE_REGIONSERVER: {
yarn.role.priority: 2,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/b0192716/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
--
diff --git
a/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
b/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
index 7377dd3..fed0bc2 100644
---
a/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
+++
b/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
@@ -563,6 +563,7 @@ public class CoreFileSystem {
public MapString, Path listPersistentInstances() throws IOException {
FileSystem fs = getFileSystem();
Path path = new Path(getBaseApplicationPath(),
SliderKeys.CLUSTER_DIRECTORY);
+log.debug(Looking for all persisted application at {}, path.toString());
if (!fs.exists(path)) {
// special case: no instances have ever been created
return new HashMapString, Path(0);
@@ -579,6 +580,9 @@ public class CoreFileSystem {
if (fs.exists(internalJson)) {
// success = this is an instance
instances.put(child.getName(), child);
+} else {
+ log.info(Malformed cluster found at {}. It does not appear to be a
valid persisted instance.,
+ child.toString());
}
}
}
git commit: SLIDER-598. HBase application package for Windows need to be tested and validated
Repository: incubator-slider
Updated Branches:
refs/heads/develop b01927162 - 96c9006d4
SLIDER-598. HBase application package for Windows need to be tested and
validated
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/96c9006d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/96c9006d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/96c9006d
Branch: refs/heads/develop
Commit: 96c9006d40c397c5f0244dfbed5b4c5c9eaece7c
Parents: b019271
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Sat Nov 1 17:31:44 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Sat Nov 1 17:31:44 2014 -0700
--
app-packages/hbase-win/appConfig-default.json | 3 +
app-packages/hbase-win/metainfo.xml | 43 ++
.../hbase-win/package/scripts/hbase_rest.py | 62
.../hbase-win/package/scripts/hbase_service.py | 9 +++
.../hbase-win/package/scripts/hbase_thrift.py | 62
.../hbase-win/package/scripts/hbase_thrift2.py | 62
.../hbase-win/package/scripts/params.py | 31 --
app-packages/hbase-win/resources-default.json | 16 +
8 files changed, 268 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96c9006d/app-packages/hbase-win/appConfig-default.json
--
diff --git a/app-packages/hbase-win/appConfig-default.json
b/app-packages/hbase-win/appConfig-default.json
index 069e2b7..80d20af 100644
--- a/app-packages/hbase-win/appConfig-default.json
+++ b/app-packages/hbase-win/appConfig-default.json
@@ -15,6 +15,9 @@
site.global.monitor_protocol: http,
site.global.hbase_additional_cp: c:\\java\\lib\\tools.jar;,
site.global.java_library_path: c:\\hdp\\hadoop\\bin,
+site.global.hbase_rest_port: 17000,
+site.global.hbase_thrift_port: 9090,
+site.global.hbase_thrift2_port: 9091,
site.hbase-env.hbase_master_heapsize: 1024m,
site.hbase-env.hbase_regionserver_heapsize: 1024m,
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96c9006d/app-packages/hbase-win/metainfo.xml
--
diff --git a/app-packages/hbase-win/metainfo.xml
b/app-packages/hbase-win/metainfo.xml
index f0c5d4f..da6121d 100644
--- a/app-packages/hbase-win/metainfo.xml
+++ b/app-packages/hbase-win/metainfo.xml
@@ -41,6 +41,18 @@
nameorg.apache.slider.monitor/name
valuehttp://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/master-status/value
/export
+ export
+nameorg.apache.slider.hbase.rest/name
+
valuehttp://${HBASE_REST_HOST}:${site.global.hbase_rest_port}/value
+ /export
+ export
+nameorg.apache.slider.hbase.thrift2/name
+
valuehttp://${HBASE_THRIFT2_HOST}:${site.global.hbase_thrift2_port}/value
+ /export
+ export
+nameorg.apache.slider.hbase.thrift/name
+
valuehttp://${HBASE_THRIFT_HOST}:${site.global.hbase_thrift_port}/value
+ /export
/exports
/exportGroup
/exportGroups
@@ -80,12 +92,43 @@
component
nameHBASE_REGIONSERVER/name
categorySLAVE/category
+minInstanceCount0/minInstanceCount
commandScript
scriptscripts/hbase_regionserver.py/script
scriptTypePYTHON/scriptType
/commandScript
/component
+ component
+nameHBASE_REST/name
+categoryMASTER/category
+appExportsQuickLinks-org.apache.slider.hbase.rest/appExports
+commandScript
+ scriptscripts/hbase_rest.py/script
+ scriptTypePYTHON/scriptType
+/commandScript
+ /component
+
+ component
+nameHBASE_THRIFT/name
+categoryMASTER/category
+appExportsQuickLinks-org.apache.slider.hbase.thrift/appExports
+commandScript
+ scriptscripts/hbase_thrift.py/script
+ scriptTypePYTHON/scriptType
+/commandScript
+ /component
+
+ component
+nameHBASE_THRIFT2/name
+categoryMASTER/category
+minInstanceCount0/minInstanceCount
+appExportsQuickLinks-org.apache.slider.hbase.thrift2/appExports
+commandScript
+ scriptscripts/hbase_thrift2.py/script
+ scriptTypePYTHON/scriptType
+/commandScript
+ /component
/components
osSpecifics
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96c9006d/app-packages/hbase-win/package/scripts/hbase_rest.py
git commit: SLIDER-598. HBase application package for Windows
Repository: incubator-slider
Updated Branches:
refs/heads/develop 9c2080baa - 48d6bd2d6
SLIDER-598. HBase application package for Windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/48d6bd2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/48d6bd2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/48d6bd2d
Branch: refs/heads/develop
Commit: 48d6bd2d62716b77210506bd07bd9754b26658ce
Parents: 9c2080b
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Fri Oct 31 19:04:40 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Fri Oct 31 19:04:40 2014 -0700
--
app-packages/hbase-win/README.txt | 2 +-
app-packages/hbase-win/appConfig-default.json | 4 ++--
app-packages/hbase-win/package/scripts/hbase.py | 5 -
app-packages/hbase-win/package/scripts/hbase_service.py | 1 -
app-packages/hbase/appConfig-default.json | 2 +-
app-packages/hbase/appConfig-secured-default.json | 2 +-
slider-agent/src/main/python/agent/ActionQueue.py | 3 ++-
.../libraries/functions/check_process_status.py | 7 ++-
8 files changed, 13 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/48d6bd2d/app-packages/hbase-win/README.txt
--
diff --git a/app-packages/hbase-win/README.txt
b/app-packages/hbase-win/README.txt
index 4e0e30a..a413e76 100644
--- a/app-packages/hbase-win/README.txt
+++ b/app-packages/hbase-win/README.txt
@@ -27,7 +27,7 @@ with appropriate parameters.
Command:
mvn clean package -Phbase-app-package-win -Dpkg.version=version
- -Dpkg.name=file name of app tarball -Dpkg.src=folder location where the
pkg is available
+ -Dpkg.name=file name of app zip file -Dpkg.src=folder location where the
pkg is available
Example:
mvn clean package -Phbase-app-package -Dpkg.version=0.98.5-hadoop2
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/48d6bd2d/app-packages/hbase-win/appConfig-default.json
--
diff --git a/app-packages/hbase-win/appConfig-default.json
b/app-packages/hbase-win/appConfig-default.json
index a802fed..069e2b7 100644
--- a/app-packages/hbase-win/appConfig-default.json
+++ b/app-packages/hbase-win/appConfig-default.json
@@ -14,12 +14,12 @@
site.global.user_group: hadoop,
site.global.monitor_protocol: http,
site.global.hbase_additional_cp: c:\\java\\lib\\tools.jar;,
-site.global.java_library_path:
c:\\hdp\\hadoop-2.4.0.2.1.3.0-1990\\bin,
+site.global.java_library_path: c:\\hdp\\hadoop\\bin,
site.hbase-env.hbase_master_heapsize: 1024m,
site.hbase-env.hbase_regionserver_heapsize: 1024m,
site.hbase-site.hbase.rootdir: ${DEFAULT_DATA_DIR},
-site.hbase-site.hbase.superuser: yarn,
+site.hbase-site.hbase.superuser: hadoop,
site.hbase-site.hbase.tmp.dir: ${AGENT_WORK_ROOT}/work/app/tmp,
site.hbase-site.hbase.local.dir: ${hbase.tmp.dir}/local,
site.hbase-site.hbase.zookeeper.quorum: ${ZK_HOST},
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/48d6bd2d/app-packages/hbase-win/package/scripts/hbase.py
--
diff --git a/app-packages/hbase-win/package/scripts/hbase.py
b/app-packages/hbase-win/package/scripts/hbase.py
index 5ae64eb..0962149 100644
--- a/app-packages/hbase-win/package/scripts/hbase.py
+++ b/app-packages/hbase-win/package/scripts/hbase.py
@@ -28,20 +28,15 @@ def hbase(name=None # 'master' or 'regionserver'
import params
Directory( params.conf_dir,
- owner = params.hbase_user,
- group = params.user_group,
recursive = True,
content = params.input_conf_files_dir
)
Directory (params.tmp_dir,
- owner = params.hbase_user,
recursive = True
)
Directory (os.path.join(params.local_dir, jars),
- owner = params.hbase_user,
- group = params.user_group,
recursive = True
)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/48d6bd2d/app-packages/hbase-win/package/scripts/hbase_service.py
--
diff --git a/app-packages/hbase-win/package/scripts/hbase_service.py
b/app-packages/hbase-win/package/scripts/hbase_service.py
index c02526c..4a1f103 100644
--- a/app-packages/hbase-win/package/scripts/hbase_service.py
+++ b/app-packages/hbase-win/package/scripts/hbase_service.py
@@ -41,7 +41,6 @@ def hbase_service(
process_cmd =
git commit: SLIDER-598. HBase application package for Windows
Repository: incubator-slider
Updated Branches:
refs/heads/develop b0a5bb088 - a07eaf0a0
SLIDER-598. HBase application package for Windows
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/a07eaf0a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/a07eaf0a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/a07eaf0a
Branch: refs/heads/develop
Commit: a07eaf0a08a6f18ea7672403e369def3080fab89
Parents: b0a5bb0
Author: Sumit Mohanty smoha...@hortonworks.com
Authored: Fri Oct 31 22:02:36 2014 -0700
Committer: Sumit Mohanty smoha...@hortonworks.com
Committed: Fri Oct 31 22:02:36 2014 -0700
--
app-packages/hbase/appConfig-default.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/a07eaf0a/app-packages/hbase/appConfig-default.json
--
diff --git a/app-packages/hbase/appConfig-default.json
b/app-packages/hbase/appConfig-default.json
index a36a693..52587e5 100644
--- a/app-packages/hbase/appConfig-default.json
+++ b/app-packages/hbase/appConfig-default.json
@@ -8,7 +8,7 @@
java_home: /usr/jdk64/jdk1.7.0_67,
system_configs: core-site,
-site.global.app_user: ${USER_NAME},
+site.global.app_user: yarn,
site.global.app_root:
${AGENT_WORK_ROOT}/app/install/hbase-${pkg.version},
site.global.ganglia_server_host: ${NN_HOST},
git commit: SLIDER-576. Default packages have a hard-coded build version - instead they need to use mvn parameter
Repository: incubator-slider Updated Branches: refs/heads/develop 196466d0e - 6ae705ac6 SLIDER-576. Default packages have a hard-coded build version - instead they need to use mvn parameter Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6ae705ac Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6ae705ac Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6ae705ac Branch: refs/heads/develop Commit: 6ae705ac6f7597609958612ec34899f8e3ad33c1 Parents: 196466d Author: Sumit Mohanty smoha...@hortonworks.com Authored: Wed Oct 29 09:43:28 2014 -0700 Committer: Sumit Mohanty smoha...@hortonworks.com Committed: Wed Oct 29 09:43:28 2014 -0700 -- app-packages/hbase/src/assembly/hbase.xml | 7 +++ app-packages/storm/src/assembly/storm.xml | 7 +++ 2 files changed, 14 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6ae705ac/app-packages/hbase/src/assembly/hbase.xml -- diff --git a/app-packages/hbase/src/assembly/hbase.xml b/app-packages/hbase/src/assembly/hbase.xml index a94c827..a74304c 100644 --- a/app-packages/hbase/src/assembly/hbase.xml +++ b/app-packages/hbase/src/assembly/hbase.xml @@ -36,6 +36,12 @@ fileMode0755/fileMode /file file + sourceappConfig-secured-default.json/source + outputDirectory//outputDirectory + filteredtrue/filtered + fileMode0755/fileMode +/file +file sourcemetainfo.xml/source outputDirectory//outputDirectory filteredtrue/filtered @@ -58,6 +64,7 @@ excludesrc/**/exclude excludetarget/**/exclude excludeappConfig-default.json/exclude +excludeappConfig-secured-default.json/exclude excludemetainfo.xml/exclude /excludes fileMode0755/fileMode http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6ae705ac/app-packages/storm/src/assembly/storm.xml -- diff --git a/app-packages/storm/src/assembly/storm.xml b/app-packages/storm/src/assembly/storm.xml index cc000c5..f7dcf13 100644 --- a/app-packages/storm/src/assembly/storm.xml +++ b/app-packages/storm/src/assembly/storm.xml @@ -36,6 +36,12 @@ fileMode0755/fileMode /file file + sourceappConfig-secured-default.json/source + outputDirectory//outputDirectory + filteredtrue/filtered + fileMode0755/fileMode +/file +file sourcemetainfo.xml/source outputDirectory//outputDirectory filteredtrue/filtered @@ -58,6 +64,7 @@ excludesrc/**/exclude excludetarget/**/exclude excludeappConfig-default.json/exclude +excludeappConfig-secured-default.json/exclude excludemetainfo.xml/exclude /excludes fileMode0755/fileMode
