SLIDER-412 Change hbase-app-package profile for HBase app package to use local HBase tar ball (Sumit and Ted)
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/809dc786 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/809dc786 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/809dc786 Branch: refs/heads/feature/SLIDER-86-windows Commit: 809dc7860933a383b294b523011162c0872e167b Parents: 65d17d8 Author: tedyu <yuzhih...@gmail.com> Authored: Tue Sep 9 11:27:36 2014 -0700 Committer: tedyu <yuzhih...@gmail.com> Committed: Tue Sep 9 11:27:36 2014 -0700 ---------------------------------------------------------------------- app-packages/hbase/README.txt | 43 ++-- app-packages/hbase/appConfig.json | 4 +- app-packages/hbase/metainfo.xml | 4 +- app-packages/hbase/pom.xml | 317 +++++++++++++++---------- app-packages/hbase/src/assembly/hbase.xml | 16 +- app-packages/storm/README.txt | 20 +- 6 files changed, 227 insertions(+), 177 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/809dc786/app-packages/hbase/README.txt ---------------------------------------------------------------------- diff --git a/app-packages/hbase/README.txt b/app-packages/hbase/README.txt index 1d5c4bb..d0e3067 100644 --- a/app-packages/hbase/README.txt +++ b/app-packages/hbase/README.txt @@ -22,10 +22,26 @@ work well as the default configuration for Slider apps. So it is advisable that when you create an application package for Slider, include sample/default resources.json and appConfig.json for a minimal Yarn cluster. -OPTION-I: Use mvn command -OPTION-II: Manual +OPTION-I: Use a downloaded tarball +OPTION-II: Use the tarball from the local repository -****** OPTION - I (use mvn command) ** +****** OPTION - I ** +To create the app package you will need the HBase tarball and invoke mvn command +with appropriate parameters. + +Command: +mvn clean package -Phbase-app-package -Dpkg.version=<version> + -Dpkg.name=<file name of app tarball> -Dpkg.src=<folder location where the pkg is available> + +Example: +mvn clean package -Phbase-app-package -Dpkg.version=0.98.5-hadoop2 + -Dpkg.name=hbase-0.98.5-hadoop2-bin.tar.gz + -Dpkg.src=/Users/user1/Downloads/0.98.5-hadoop2 + +App package can be found in + app-packages/hbase/target/slider-hbase-app-package-${pkg.version}.zip + +****** OPTION - II ** You need the HBase version available on local maven repo to create the Slider App Package for HBase. Download the tarball for HBase: @@ -47,29 +63,12 @@ After HBase tarball is published locally in maven repository, you can use the fo App package can be found in app-packages/hbase/target/apache-slider-hbase-${hbase.version}-app-package-${slider.version}.zip -Verify the content using - zip -Tv apache-slider-hbase-*.zip - If an HBase version older than 0.98.3 is desired, it must be installed in the local maven repo. A less descriptive file name can be specified with -Dapp.package.name=HBase_98dot3 which would create a file HBase_98dot3.zip. -****** OPTION - II (manual) ** -The Slider App Package for HBase can also be created manually. - -Download the tarball for HBase: - e.g. path to tarball ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz - -Copy the hbase tarball to package/files - cp ~/Downloads/hbase-0.98.3-hadoop2-bin.tar.gz package/files - -Edit appConfig.json/metainfo.xml - Replace 4 occurrences of "${hbase.version}" with the hbase version values such as "0.98.3-hadoop2" - Replace 1 occurrence of "${app.package.name}" with the desired app package name, e.g. "hbase-v098" - -Create a zip package at the root of the package (<slider enlistment>/app-packages/hbase/) - zip -r hbase-v098.zip . +****** Verifying the content ** Verify the content using - zip -Tv hbase-v098.zip + zip -Tv apache-slider-hbase-*.zip http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/809dc786/app-packages/hbase/appConfig.json ---------------------------------------------------------------------- diff --git a/app-packages/hbase/appConfig.json b/app-packages/hbase/appConfig.json index 07d1b4e..3743602 100644 --- a/app-packages/hbase/appConfig.json +++ b/app-packages/hbase/appConfig.json @@ -3,13 +3,13 @@ "metadata": { }, "global": { - "application.def": "${app.package.name}.zip", + "application.def": "slider-hbase-app-package-${bld.version}.zip", "create.default.zookeeper.node": "true", "java_home": "/usr/jdk64/jdk1.7.0_45", "system_configs": "core-site", "site.global.app_user": "yarn", - "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-${hbase.version}", + "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-${pkg.version}", "site.global.ganglia_server_host": "${NN_HOST}", "site.global.ganglia_server_port": "8667", http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/809dc786/app-packages/hbase/metainfo.xml ---------------------------------------------------------------------- diff --git a/app-packages/hbase/metainfo.xml b/app-packages/hbase/metainfo.xml index 1445c31..10bee97 100644 --- a/app-packages/hbase/metainfo.xml +++ b/app-packages/hbase/metainfo.xml @@ -25,7 +25,7 @@ 1. Ensure parent dir for path (hbase-site/hbase.rootdir) is accessible to the App owner. 2. Ensure ZK root (hbase-site/zookeeper.znode.parent) is unique for the App instance. </comment> - <version>${hbase.version}</version> + <version>${pkg.version}</version> <type>YARN-APP</type> <minHadoopVersion>2.1.0</minHadoopVersion> <exportedConfigs>hbase-site</exportedConfigs> @@ -160,7 +160,7 @@ <packages> <package> <type>tarball</type> - <name>files/hbase-${hbase.version}-bin.tar.gz</name> + <name>files/hbase-${pkg.version}-bin.tar.gz</name> </package> </packages> </osSpecific> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/809dc786/app-packages/hbase/pom.xml ---------------------------------------------------------------------- diff --git a/app-packages/hbase/pom.xml b/app-packages/hbase/pom.xml index 6f1da9c..3d57ac5 100644 --- a/app-packages/hbase/pom.xml +++ b/app-packages/hbase/pom.xml @@ -1,5 +1,6 @@ <?xml version="1.0"?> -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -24,9 +25,10 @@ </parent> <modelVersion>4.0.0</modelVersion> <artifactId>slider-hbase-app-package</artifactId> - <packaging>jar</packaging> + <packaging>pom</packaging> <name>Slider HBase App Package</name> <description>Slider HBase App Package</description> + <version>${bld.version}</version> <properties> <work.dir>package-tmp</work.dir> <app.package.name>${project.artifactId}-${project.version}</app.package.name> @@ -35,10 +37,71 @@ <profiles> <profile> <id>hbase-app-package</id> + <properties> + <bld.version>${pkg.version}</bld.version> + </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <version>1.7</version> + <executions> + <execution> + <id>copy</id> + <phase>validate</phase> + <configuration> + <target name="copy and rename file"> + <copy file="${pkg.src}/${pkg.name}" tofile="${project.build.directory}/${pkg.name}"/> + </target> + </configuration> + <goals> + <goal>run</goal> + </goals> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> + <configuration> + <tarLongFileMode>gnu</tarLongFileMode> + <descriptor>src/assembly/hbase.xml</descriptor> + <appendAssemblyId>false</appendAssemblyId> + </configuration> + <executions> + <execution> + <id>build-tarball</id> + <phase>package</phase> + <goals> + <goal>single</goal> + </goals> + </execution> + </executions> + </plugin> + + </plugins> + </build> + </profile> + <profile> + <id>hbase-app-package-it</id> + <properties> + <pkg.version>${hbase.version}</pkg.version> + <bld.version>0.51.0-incubating-SNAPSHOT</bld.version> + <pkg.src>${project.build.directory}/${work.dir}</pkg.src> + <pkg.name>hbase-${hbase.version}-bin.tar.gz</pkg.name> + </properties> + <build> + <resources> + <resource> + <directory>src/test/resources</directory> + <filtering>true</filtering> + <targetPath>${project.build.directory}/test-config</targetPath> + </resource> + </resources> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptor>src/assembly/hbase.xml</descriptor> @@ -95,7 +158,8 @@ <java.awt.headless>true</java.awt.headless> <!-- this property must be supplied--> <slider.conf.dir>${slider.conf.dir}</slider.conf.dir> - <slider.bin.dir>../../slider-assembly/target/slider-${project.version}-all/slider-${project.version}</slider.bin.dir> + <slider.bin.dir>../../slider-assembly/target/slider-${project.version}-all/slider-${project.version} + </slider.bin.dir> <test.app.pkg.dir>target</test.app.pkg.dir> <test.app.pkg.file>${app.package.name}.zip</test.app.pkg.file> <test.app.resource>target/test-config/resources.json</test.app.resource> @@ -103,143 +167,134 @@ </systemPropertyVariables> </configuration> </plugin> + <plugin> + <artifactId>maven-compiler-plugin</artifactId> + <dependencies> + <dependency> + <groupId>org.codehaus.groovy</groupId> + <artifactId>groovy-eclipse-compiler</artifactId> + <version>${groovy-eclipse-compiler.version}</version> + </dependency> + <dependency> + <groupId>org.codehaus.groovy</groupId> + <artifactId>groovy-eclipse-batch</artifactId> + <version>${groovy-eclipse-batch.version}</version> + </dependency> + </dependencies> + </plugin> + + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-surefire-plugin</artifactId> + <configuration> + <!-- can't figure out how to get the surefire plugin not to pick up the ITs, so skip it entirely --> + <skip>true</skip> + </configuration> + </plugin> </plugins> </build> + <dependencies> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase</artifactId> + <version>${hbase.version}</version> + <classifier>bin</classifier> + <type>tar.gz</type> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + </dependency> + + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-minicluster</artifactId> + <scope>test</scope> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-server</artifactId> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-protocol</artifactId> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-it</artifactId> + <classifier>tests</classifier> + <exclusions> + <exclusion> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-client</artifactId> + </exclusion> + </exclusions> + <scope>test</scope> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-server</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.slider</groupId> + <artifactId>slider-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.slider</groupId> + <artifactId>slider-funtest</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.codehaus.groovy</groupId> + <artifactId>groovy-all</artifactId> + <scope>test</scope> + </dependency> + </dependencies> </profile> </profiles> <build> - <!-- resources are filtered for dynamic updates. This gets build info in--> - <resources> - <resource> - <directory>src/test/resources</directory> - <filtering>true</filtering> - <targetPath>${project.build.directory}/test-config</targetPath> - </resource> - </resources> - - <plugins> - <plugin> - <artifactId>maven-compiler-plugin</artifactId> - <dependencies> - <dependency> - <groupId>org.codehaus.groovy</groupId> - <artifactId>groovy-eclipse-compiler</artifactId> - <version>${groovy-eclipse-compiler.version}</version> - </dependency> - <dependency> - <groupId>org.codehaus.groovy</groupId> - <artifactId>groovy-eclipse-batch</artifactId> - <version>${groovy-eclipse-batch.version}</version> - </dependency> - </dependencies> - </plugin> - - <plugin> - <groupId>org.apache.maven.plugins</groupId> - <artifactId>maven-surefire-plugin</artifactId> - <configuration> - <!-- can't figure out how to get the surefire plugin not to pick up the ITs, so skip it entirely --> - <skip>true</skip> - </configuration> - </plugin> - </plugins> </build> <dependencies> - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase</artifactId> - <version>${hbase.version}</version> - <classifier>bin</classifier> - <type>tar.gz</type> - </dependency> - <dependency> - <groupId>junit</groupId> - <artifactId>junit</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-client</artifactId> - </dependency> - - <dependency> - <groupId>org.apache.hadoop</groupId> - <artifactId>hadoop-minicluster</artifactId> - <scope>test</scope> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-server</artifactId> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-protocol</artifactId> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-common</artifactId> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-common</artifactId> - <classifier>tests</classifier> - <scope>test</scope> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-it</artifactId> - <classifier>tests</classifier> - <exclusions> - <exclusion> - <groupId>org.apache.hadoop</groupId> - <artifactId>hadoop-client</artifactId> - </exclusion> - </exclusions> - <scope>test</scope> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-hadoop-compat</artifactId> - <classifier>tests</classifier> - <scope>test</scope> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-hadoop2-compat</artifactId> - <classifier>tests</classifier> - <scope>test</scope> - </dependency> - - <dependency> - <groupId>org.apache.hbase</groupId> - <artifactId>hbase-server</artifactId> - <classifier>tests</classifier> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.apache.slider</groupId> - <artifactId>slider-core</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.apache.slider</groupId> - <artifactId>slider-funtest</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.codehaus.groovy</groupId> - <artifactId>groovy-all</artifactId> - <scope>test</scope> - </dependency> </dependencies> </project> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/809dc786/app-packages/hbase/src/assembly/hbase.xml ---------------------------------------------------------------------- diff --git a/app-packages/hbase/src/assembly/hbase.xml b/app-packages/hbase/src/assembly/hbase.xml index ff1c395..e62da84 100644 --- a/app-packages/hbase/src/assembly/hbase.xml +++ b/app-packages/hbase/src/assembly/hbase.xml @@ -41,6 +41,12 @@ <filtered>true</filtered> <fileMode>0755</fileMode> </file> + <file> + <source>${pkg.src}/${pkg.name}</source> + <outputDirectory>package/files</outputDirectory> + <filtered>false</filtered> + <fileMode>0755</fileMode> + </file> </files> <fileSets> @@ -58,15 +64,5 @@ <directoryMode>0755</directoryMode> </fileSet> - <fileSet> - <directory>${project.build.directory}/${work.dir}</directory> - <outputDirectory>package/files</outputDirectory> - <includes> - <include>hbase-${hbase.version}-bin.tar.gz</include> - </includes> - <fileMode>0755</fileMode> - <directoryMode>0755</directoryMode> - </fileSet> - </fileSets> </assembly> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/809dc786/app-packages/storm/README.txt ---------------------------------------------------------------------- diff --git a/app-packages/storm/README.txt b/app-packages/storm/README.txt index 288604a..df48305 100644 --- a/app-packages/storm/README.txt +++ b/app-packages/storm/README.txt @@ -17,19 +17,19 @@ How to create a Slider app package for Storm? -To create the app package you will need the Storm tarball copied to a specific location. -Various configurations provided in this sample are customized for apache-storm-0.9.1.2.1.1.0-237.tar.gz. -So if you use a different version you may need to edit a few config values. +To create the app package you will need the Storm tarball and invoke mvn command +with appropriate parameters. -Replace the placeholder tarball for Storm. - cp ~/Downloads/apache-storm-0.9.3.0.2.5.0-100.tar.gz package/files/ - rm package/files/apache-storm-0.9.1.2.1.1.0-237.tar.gz.REPLACE +Command: +mvn clean package -Pstorm-app-package -Dpkg.version=<version> + -Dpkg.name=<file name of app tarball> -Dpkg.src=<folder location where the pkg is available> -Create a zip package at the root of the package (<slider enlistment>/app-packages/storm-v0_91/) - zip -r Apache_Storm_v_0_9_3.zip . +Example: +mvn clean package -Pstorm-app-package -Dpkg.version=0.9.3.2.2.0.0-578 + -Dpkg.name=apache-storm-0.9.3.2.2.0.0-578.tar.gz -Dpkg.src=/Users/user1/Downloads -Verify the content using - unzip -l "$@" Apache_Storm_v_0_9_3.zip +App package can be found in + app-packages/storm/target/slider-storm-app-package-${pkg.version}.zip While appConfig.json and resources.json are not required for the package they work well as the default configuration for Slider apps. So its advisable that when you
