+1, binding
Review process
# symlink /usr/local/bin/protoc to the homebrew installed 2.5.0 version
# delete all 2.1.0-beta artifacts in the mvn repo:
find ~/.m2 -name 2.1.0-beta -print | xargs rm -rf
# checkout hbase apache/branch-0.95 (commit # b58d596 )
# switch to ASF repo (arun's private repo is in the POM, with JARs with the
same sha1 sum, I'm just being rigorous)
<repository>
<id>ASF Staging</id>
<url>https://repository.apache.org/content/groups/staging/</url>
</repository>
# clean build of hbase tar against the beta artifacts
mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true
-Dhadoop.profile=2.0 -Dhadoop-two.version=2.1.0-beta
# Observe DL taking place
]
[INFO] --- maven-assembly-plugin:2.4:single (default-cli) @ hbase ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly
parameter.
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building HBase - Common 0.95.3-SNAPSHOT
[INFO]
------------------------------------------------------------------------
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom(2
KB at 3.3 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom(34
KB at 383.8 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom(17
KB at 184.0 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom(26
KB at 293.3 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom(17
KB at 176.2 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom(7
KB at 79.0 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar(17
KB at 146.1 KB/sec)
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar(47
KB at 241.7 KB/sec)
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar(2657
KB at 2260.9 KB/sec)
[INFO]
# get md5 sum of hadoop-common-2.1.0-beta artifact in
https://repository.apache.org/content/groups/staging/ :
0166f5c94d3699b3a37efc16ebb1ceea3acb3b53
# verify version of artifact in local m2 repo
$ sha1sum
~/.m2/repository/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
0166f5c94d3699b3a37efc16ebb1ceea3acb3b53
# in hbase/hbase-assembly/target , gunzip then untar the
hbase-0.95.3-SNAPSHOT-bin.tar file
# Patch the Hoya POM to use 2.1.0-beta instead of a local 2.1.1-SNAPSHOT
# run some of the hbase cluster deploy & flexing tests
mvn clean test -Pstaging
(all tests pass after 20 min)
== functional tests =
# build and the Hoya JAR with classpath
mvn package -Pstaging
# D/L the binary .tar.gz file, and scp to an ubuntu VM with the hadoop conf
properties for net-accessible HDFS & YARN services, no memory limits on
containers
https://github.com/hortonworks/hoya/tree/master/src/test/configs/ubuntu
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/core-site.xml
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/yarn-site.xml
# stop the running hadoop-2.1.1-snapshot cluster
# start the new cluster services
hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
# Verify in dfshealth that NN is up, version is built ( 2013-08-15T20:48Z
by hortonmu from branch-2.1.0-beta ). (the NN came up in safe mode as a
couple of in-flight blocks were missing; NN recovered from this happily
after 20s)
# copy hbase-0.93 tarball to HDFS
hdfs dfs -copyFromLocal hbase-0.95.3-SNAPSHOT-bin.tar
hdfs://ubuntu:9000/hbase.tar
# unfreeze an hbase cluster that was running on hbase-0.95.2 & Hadoop-2.1.1
(protobuf 2.4) versions & set up to use hdfs://ubuntu:9000/hbase.tar as the
hbase image to install
java -jar target/hoya-0.3-SNAPSHOT.jar org.apache.hadoop.hoya.Hoya thaw
cl1 --manager ubuntu:8032 --filesystem hdfs://ubuntu:9000
# verify cluster is running according to Hoya CLI and YARN web GUI
# point browser at HDFS master: http://ubuntu:8080/master-status , verify
it is happy
# fault injection: ssh-in and kill -9 all HRegionServers
# verify that within 60s the # no. of region servers is as desired; YARN
notified Hoya of container loss; replacement containers requested and
region servers deployed.
# Freeze the cluster:
java -jar target/hoya-0.3-SNAPSHOT.jar org.apache.hadoop.hoya.Hoya\
freeze cl1 \
--manager ubuntu:8032 --filesystem hdfs://ubuntu:9000
# verify that hoya list command and GUI show app as finished; HBase cluster
is no longer present
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
