[
https://issues.apache.org/jira/browse/PHOENIX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
alex kamil updated PHOENIX-1041:
--------------------------------
Description:
Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH
deployments are very common and it's a nightmare to debug , e.g. in case hadoop
version in phoenix doesn't match the one deployed on the cluster (e.g. see
PHOENIX-56 - Phoenix/JDBC becomes unresponsive and PHOENIX-417 - Hadoop 2 woes)
This example is for phoenix 3.1 and CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and
hbase-2.0.0-cdh4.6.0
1. phoenix/pom.xml changes:
add cloudera repository
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
2. update hadoop.profile version
<!-- Hadoop Versions -->
<hadoop-one.version>1.0.4</hadoop-one.version>
<!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
<hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
3. update dependency versions
<!-- Dependency versions -->
<!--hbase.version>0.94.15</hbase.version-->
<hbase.version>0.94.15-cdh4.6.0</hbase.version>
<!--hadoop.version>2.0.0</hadoop.version-->
<hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
4. build phoenix with hadoop2 profile
mvn clean install -DskipTests -Dhadoop.profile=2
5. stop hadoop and hbase cluster if running
6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each
hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
7. add phoenix-core jar to hbase classpath
update /etc/hbase/conf/hbase-env.sh:
export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
(Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error, this
is avoided by loading phoenix jar before the rest of the hbase dependencies
with HBASE_CLASSPATH_PREFIX)
8. (optional) add phoenix-core jar to hadoop classpath
update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf directory
if doesn't exist)
export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
9. start hadoop/hbase cluster
check classpath:
hbase classpath|tr ":" "\n"|grep phoenix
create a test table with hbase shell:
hbase shell
hbase>create 't1',{NAME => 'f1',VERSIONS => 5}
check hbase/hadoop/zk logs for errors
start sqlline:
10. enable DEBUG logging in sqlline log4j.properties
(otherwise sqlline is just "hanging"in case of hadoop version mismatch and
hides the related error: java.lang.NoSuchMethodError:
org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream,
java.io.IOException: Could not set up IO Streams,
client.HConnectionManager$HConnectionImplementation: getMaster attempt 8 of 14
failed )
>save a backup copy of sqlline log4j.properties
>replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
11. add commons-collections, hadoop-common, hadoop-auth and hbase jars from
CDH to sqlline classpath
(Otherwise getting java.lang.NoClassDefFoundError:
org/apache/commons/collections/map/UnmodifiableMap exception)
update phoenix/bin/sqlline.py:
extrajars=
"/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar"
java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+
phoenix_utils.phoenix_client_jar + \
12. add /etc/hbase/conf/hbase-site.xml from one of the hbase nodes to
phoenix/bin/
13. test
phoenix/bin/sqlline.py <my_zookeeper_host>
14. building JDBC java client
>same jars as in step 11 need to be added to the classpath of a phoenix JDBC
>client:
/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar
>enable DEBUG logging:
add log4j.properties file to /src/main/resources/
add to main:
import org.apache.log4j.Logger;
import org.apache.log4j.Level;
..
Logger.getRootLogger().setLevel(Level.DEBUG);
>add hbase-site.xml from one of the hbase nodes to to /src/main/resources/
>if building with gradle:
add to build.gradle:
repositories { maven { url
"https://repository.cloudera.com/artifactory/cloudera-repos/"; }
}
compile(group: 'org.apache.hadoop', name:'hadoop-common',version:'2.0.0-alpha')
compile(group: 'org.apache.hadoop', name: 'hadoop-auth', version:'2.0.0-alpha')
compile(group: 'org.apache.hbase', name: 'hbase', version:'0.94.15')
Note the naming: it's hadoop-common-2.0.0-alpha, not
hadoop-common-2.0.0-cdh4.6.0, see
http://mvnrepository.com/artifact/org.apache.hadoop
was:
Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH
deployments are very common and it's a nightmare to debug , e.g. in case hadoop
version in phoenix doesn't match the one deployed on the cluster (e.g. see
PHOENIX-56 - Phoenix/JDBC becomes unresponsive and PHOENIX-417 - Hadoop 2 woes)
This example is for phoenix 3.1 and CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and
hbase-2.0.0-cdh4.6.0
1. phoenix/pom.xml changes:
add cloudera repository
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
2. update hadoop.profile version
<!-- Hadoop Versions -->
<hadoop-one.version>1.0.4</hadoop-one.version>
<!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
<hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
3. update dependency versions
<!-- Dependency versions -->
<!--hbase.version>0.94.15</hbase.version-->
<hbase.version>0.94.15-cdh4.6.0</hbase.version>
<!--hadoop.version>2.0.0</hadoop.version-->
<hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
4. build phoenix with hadoop2 profile
mvn clean install -DskipTests -Dhadoop.profile=2
5. stop hadoop and hbase cluster if running
6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each
hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
7. add phoenix-core jar to hbase classpath
update /etc/hbase/conf/hbase-env.sh:
export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
(Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error, this
is avoided by loading phoenix jar before the rest of the hbase dependencies
with HBASE_CLASSPATH_PREFIX)
8. (optional) add phoenix-core jar to hadoop classpath
update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf directory
if doesn't exist)
export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
9. start hadoop/hbase cluster
check classpath:
hbase classpath|tr ":" "\n"|grep phoenix
create a test table with hbase shell:
hbase shell
hbase>create 't1',{NAME => 'f1',VERSIONS => 5}
check hbase/hadoop/zk logs for errors
start sqlline:
10. enable DEBUG logging in sqlline log4j.properties
(otherwise sqlline is just "hanging"in case of hadoop version mismatch and
hides the related error: java.lang.NoSuchMethodError:
org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream,
java.io.IOException: Could not set up IO Streams,
client.HConnectionManager$HConnectionImplementation: getMaster attempt 8 of 14
failed )
>save a backup copy of sqlline log4j.properties
>replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
11. add commons-collections, hadoop-common, hadoop-auth and hbase jars from
CDH to sqlline classpath
(Otherwise getting java.lang.NoClassDefFoundError:
org/apache/commons/collections/map/UnmodifiableMap exception)
update phoenix/bin/sqlline.py:
extrajars=
"/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar"
java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+
phoenix_utils.phoenix_client_jar + \
12. add /etc/hbase/conf/hbase-site.xml from one of the hbase nodes to
phoenix/bin/
13. test
phoenix/bin/sqlline.py <my_zookeeper_host>
14. building JDBC java client
>same jars as in step 11 need to be added to the classpath of a phoenix JDBC
>client:
/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar
>enable DEBUG logging:
add log4j.properties file to /src/main/resources/
add to main:
import org.apache.log4j.Logger;
import org.apache.log4j.Level;
..
Logger.getRootLogger().setLevel(Level.DEBUG);
>add hbase-site.xml from one of the hbase nodes to to /src/main/resources/
>if building with gradle:
add to build.gradle:
repositories { maven { url
"https://repository.cloudera.com/artifactory/cloudera-repos/"; }
}
compile(group: 'org.apache.hadoop', name:'hadoop-common',version:'2.0.0-alpha')
compile(group: 'org.apache.hadoop', name: 'hadoop-auth', version:'2.0.0-alpha')
compile(group: 'org.apache.hbase', name: 'hbase', version:'0.94.15')
Note the naming: it's hadoop-common-2.0.0-alpha, not
hadoop-common-2.0.0-cdh4.6.0, if you use hadoop-common-2.0.0-cdh4.6.0, gradle
will complain on unresolved dependencies during build, get these jars from
http://mvnrepository.com/artifact/org.apache.hadoop
> Building phoenix with hadoop/hbase from CDH
> -------------------------------------------
>
> Key: PHOENIX-1041
> URL: https://issues.apache.org/jira/browse/PHOENIX-1041
> Project: Phoenix
> Issue Type: Improvement
> Reporter: alex kamil
> Priority: Minor
>
> Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH
> deployments are very common and it's a nightmare to debug , e.g. in case
> hadoop version in phoenix doesn't match the one deployed on the cluster (e.g.
> see PHOENIX-56 - Phoenix/JDBC becomes unresponsive and PHOENIX-417 - Hadoop 2
> woes)
> This example is for phoenix 3.1 and CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and
> hbase-2.0.0-cdh4.6.0
> 1. phoenix/pom.xml changes:
> add cloudera repository
> <repository>
> <id>cloudera</id>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> 2. update hadoop.profile version
> <!-- Hadoop Versions -->
> <hadoop-one.version>1.0.4</hadoop-one.version>
> <!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
> <hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
> 3. update dependency versions
> <!-- Dependency versions -->
> <!--hbase.version>0.94.15</hbase.version-->
> <hbase.version>0.94.15-cdh4.6.0</hbase.version>
> <!--hadoop.version>2.0.0</hadoop.version-->
> <hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
> 4. build phoenix with hadoop2 profile
> mvn clean install -DskipTests -Dhadoop.profile=2
> 5. stop hadoop and hbase cluster if running
> 6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each
> hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
> 7. add phoenix-core jar to hbase classpath
> update /etc/hbase/conf/hbase-env.sh:
>
>
> export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
> (Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error,
> this is avoided by loading phoenix jar before the rest of the hbase
> dependencies with HBASE_CLASSPATH_PREFIX)
> 8. (optional) add phoenix-core jar to hadoop classpath
> update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf
> directory if doesn't exist)
> export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
> 9. start hadoop/hbase cluster
> check classpath:
> hbase classpath|tr ":" "\n"|grep phoenix
> create a test table with hbase shell:
> hbase shell
> hbase>create 't1',{NAME => 'f1',VERSIONS => 5}
> check hbase/hadoop/zk logs for errors
> start sqlline:
> 10. enable DEBUG logging in sqlline log4j.properties
> (otherwise sqlline is just "hanging"in case of hadoop version mismatch and
> hides the related error: java.lang.NoSuchMethodError:
> org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream,
> java.io.IOException: Could not set up IO Streams,
> client.HConnectionManager$HConnectionImplementation: getMaster attempt 8 of
> 14 failed )
> >save a backup copy of sqlline log4j.properties
> >replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
> 11. add commons-collections, hadoop-common, hadoop-auth and hbase jars
> from CDH to sqlline classpath
> (Otherwise getting java.lang.NoClassDefFoundError:
> org/apache/commons/collections/map/UnmodifiableMap exception)
> update phoenix/bin/sqlline.py:
> extrajars=
> "/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
> cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar"
>
>
> java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+
> phoenix_utils.phoenix_client_jar + \
> 12. add /etc/hbase/conf/hbase-site.xml from one of the hbase nodes to
> phoenix/bin/
> 13. test
> phoenix/bin/sqlline.py <my_zookeeper_host>
> 14. building JDBC java client
> >same jars as in step 11 need to be added to the classpath of a phoenix JDBC
> >client:
> /usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
> cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar
> >enable DEBUG logging:
> add log4j.properties file to /src/main/resources/
> add to main:
> import org.apache.log4j.Logger;
> import org.apache.log4j.Level;
> ..
> Logger.getRootLogger().setLevel(Level.DEBUG);
> >add hbase-site.xml from one of the hbase nodes to to /src/main/resources/
> >if building with gradle:
> add to build.gradle:
> repositories { maven { url
> "https://repository.cloudera.com/artifactory/cloudera-repos/"; }
> }
> compile(group: 'org.apache.hadoop',
> name:'hadoop-common',version:'2.0.0-alpha')
> compile(group: 'org.apache.hadoop', name: 'hadoop-auth',
> version:'2.0.0-alpha')
> compile(group: 'org.apache.hbase', name: 'hbase', version:'0.94.15')
> Note the naming: it's hadoop-common-2.0.0-alpha, not
> hadoop-common-2.0.0-cdh4.6.0, see
> http://mvnrepository.com/artifact/org.apache.hadoop
--
This message was sent by Atlassian JIRA
(v6.2#6252)
