[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954099#comment-14954099 ] Hans Zeller commented on TRAFODION-1521: The fix will do several things: To address issue #1, it will add a new script, core/sqf/sql/scripts/get_libhdfs_files. The new script will download Google Protocol Buffers 2.5.0 and a source tar file for Hadoop and build protobuf-2.5.0 and the Hadoop native libraries. It will then copy the libraries (libhadoop.so and libhdfs.so) to $MY_SQROOT/export/lib${SQ_MBTYPE}. It will also copy the include file hdfs.h to $MY_SQROOG/include. Since this step is very time-consuming, it will take a shortcut if Hadoop is installed (e.g. with install_local_hadoop), in that case it will copy the existing native libraries. To address the remaining needs for jar files during the build, the hbasetmlib2 build will be converted to Maven. We will also remove the case in sqenvcom.sh where a configuration does not have install_local_hadoop, Cloudera, Hortonworks or MapR installed, but has a TOOLSDIR. Such cases should use the recently added case for a native Apache Hadoop installation or the case where no HBase installation is found. That last case will print out a reminder that can be ignored. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954103#comment-14954103 ] Hans Zeller commented on TRAFODION-1521: Note: Once this fix is checked in, users will need to do "sqgen" to update their classpath. This is because we need a new jar in the classpath: trafodion-dtm-1.2.0.jar. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955087#comment-14955087 ] ASF GitHub Bot commented on TRAFODION-1521: --- GitHub user zellerh opened a pull request: https://github.com/apache/incubator-trafodion/pull/118 TRAFODION-1521 Build Trafodion without having HBase installed Adding a new script, core/sqf/sql/scripts/get_libhdfs_files. The new script will download Google Protocol Buffers 2.5.0 and a source tar file for Hadoop and build protobuf-2.5.0 and the Hadoop native libraries. It will then copy the libraries (libhadoop.so and libhdfs.so) to $MY_SQROOT/export/lib$ {SQ_MBTYPE}. It will also copy the include file hdfs.h to $MY_SQROOT/include. Since this step is very time-consuming, it takes a shortcut if Hadoop is installed (e.g. with install_local_hadoop), in that case it will copy the existing native libraries. To address the remaining needs for jar files during the build, the hbasetmlib2 build is converted to Maven. We also remove the case in sqenvcom.sh where a configuration does not have install_local_hadoop, Cloudera, Hortonworks or MapR installed, but has a TOOLSDIR. Such cases should use the recently added case for a native Apache Hadoop installation or the case where no HBase installation is found. That last case will print out a reminder that can be ignored. Note: Users will need to do "sqgen" to update their classpath. This is because we need a new jar in the classpath: trafodion-dtm-1.2.0.jar. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zellerh/incubator-trafodion bug/1521 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/118.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #118 commit 77eab6badc3cebaef392a03b25c3d9e1dc5106c2 Author: Hans Zeller Date: 2015-10-13T14:40:46Z TRAFODION-1521 Build Trafodion without having HBase installed Adding a new script, core/sqf/sql/scripts/get_libhdfs_files. The new script will download Google Protocol Buffers 2.5.0 and a source tar file for Hadoop and build protobuf-2.5.0 and the Hadoop native libraries. It will then copy the libraries (libhadoop.so and libhdfs.so) to $MY_SQROOT/export/lib$ {SQ_MBTYPE}. It will also copy the include file hdfs.h to $MY_SQROOT/include. Since this step is very time-consuming, it takes a shortcut if Hadoop is installed (e.g. with install_local_hadoop), in that case it will copy the existing native libraries. To address the remaining needs for jar files during the build, the hbasetmlib2 build is converted to Maven. We also remove the case in sqenvcom.sh where a configuration does not have install_local_hadoop, Cloudera, Hortonworks or MapR installed, but has a TOOLSDIR. Such cases should use the recently added case for a native Apache Hadoop installation or the case where no HBase installation is found. That last case will print out a reminder that can be ignored. Note: Users will need to do "sqgen" to update their classpath. This is because we need a new jar in the classpath: trafodion-dtm-1.2.0.jar. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955141#comment-14955141 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41883676 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" --- End diff -- Did you mean to echo $USAGE here? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955147#comment-14955147 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41884484 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} --- End diff -- Might get an error on this if you went through the code path above that also does this "rm". > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955151#comment-14955151 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41884769 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} +tar -xzf ${PROTOBUF_TAR} >>${LOGFILE} + fi + + if [[ ! -d $PROTOBUF_TGT_ID ]]; then +cd ${PROTOBUF_ID} +echo "Building Google Protocol Buffers, this could take a while..." | tee -a ${LOGFILE} +if [[ $VERBOSE == true ]]; then + ./configure --prefix=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} 2>&1 | tee -a ${LOGFILE} +else + ./configure --prefix=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} 2>&1 >>${LOGFILE} +fi +if [[ $? != 0 ]]; then + echo "Error during configure step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +make 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then + echo "Error during make step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +# skip the tests +# make check 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then --- End diff -- Probably should comment out this "if" block too since the "make check" is commented out. Though it is harmless (the "if" should always be false). > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Report
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955165#comment-14955165 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41885829 --- Diff: core/sqf/sqenvcom.sh --- @@ -580,6 +547,13 @@ EOF echo " ERROR: Unable to determine location of HBase lib directory" fi + if [[ -d $TOOLSDIR/thrift-0.9.0 ]]; then +# this is mostly for a build environment, where we need +# thrift from TOOLSDIR +export THRIFT_LIB_DIR=$TOOLSDIR/thrift-0.9.0/lib +export THRIFT_INC_DIR=$TOOLSDIR/thrift-0.9.0/include + fi + if [ -n "$HBASE_CNF_DIR" -a -n "$HADOOP_CNF_DIR" -a \ --- End diff -- You removed Thrift from the required list in bldenvchk and set the variable only if Thrift is installed in the TOOLSDIR. What happens if Thrift is not installed? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955166#comment-14955166 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41885893 --- Diff: core/sql/nskgmake/Makerules.linux --- @@ -80,20 +80,14 @@ OBJSUFFIX := o ### # Define several directories used by various parts of the Linux build ### -NSK_SQ:= $(MY_SQROOT) -NSK := $(MY_SQROOT)/sql/nsk -XMPIROOT= $(NSK_SQ)/export/lib$(SQ_MBTYPE) -NSKBIN = $(NSK_SQ)/export/bin$(SQ_MBTYPE) -NSKROOT = $(NSK_SQ)/sql/nsk +LIBROOT = $(MY_SQROOT)/export/lib$(SQ_MBTYPE) +TRAF_BIN = $(MY_SQROOT)/export/bin$(SQ_MBTYPE) +XMPIROOT= $(LIBROOT) ifeq ($(SQ_BUILD_TYPE),release) --- End diff -- Glad to see the NSK stuff cleaned up. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955168#comment-14955168 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41886129 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files --- End diff -- Do we need the Apache header? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955172#comment-14955172 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41886366 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + --- End diff -- We have been using 2.5.? versions. Is 2.6.0 compatible with HBase .98? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955175#comment-14955175 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41886534 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + --- End diff -- Would it be useful to do : protoc --version and if the default version is 2.5.0 or greater, than remove this download step and use the installed version? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955180#comment-14955180 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41886771 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" --- End diff -- Thanks, will fix that. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955182#comment-14955182 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41886935 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} --- End diff -- The "-rf" option will suppress errors if the files don't exist. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. T
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955184#comment-14955184 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41887027 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} +tar -xzf ${PROTOBUF_TAR} >>${LOGFILE} + fi + + if [[ ! -d $PROTOBUF_TGT_ID ]]; then +cd ${PROTOBUF_ID} +echo "Building Google Protocol Buffers, this could take a while..." | tee -a ${LOGFILE} +if [[ $VERBOSE == true ]]; then + ./configure --prefix=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} 2>&1 | tee -a ${LOGFILE} +else + ./configure --prefix=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} 2>&1 >>${LOGFILE} +fi +if [[ $? != 0 ]]; then + echo "Error during configure step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +make 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then + echo "Error during make step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +# skip the tests +# make check 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then --- End diff -- Sure, will do. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Esti
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955188#comment-14955188 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41887305 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; --- End diff -- It looks like someone can set LIBHDFS_TEMP_DIR before calling this script but if they specify the -d option, tempDir is overwritten - I assume this is the desired action. Also, LOGFILE is set earlier based on LIBHDFS_TEMP_DIR, if the user changes this value here, should the LOGFILE also be changed to match? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955193#comment-14955193 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41887592 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi --- End diff -- What happens if the directory has not been created. For example, you may need sudo access to create the temp directory or the user specified some invalid location. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955195#comment-14955195 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41887745 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + --- End diff -- There is another library - libhadoop.so, so the assumption is that if libhdfs.so exists so does libhadoop.so. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955198#comment-14955198 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41888073 --- Diff: core/sqf/sqenvcom.sh --- @@ -580,6 +547,13 @@ EOF echo " ERROR: Unable to determine location of HBase lib directory" fi + if [[ -d $TOOLSDIR/thrift-0.9.0 ]]; then +# this is mostly for a build environment, where we need +# thrift from TOOLSDIR +export THRIFT_LIB_DIR=$TOOLSDIR/thrift-0.9.0/lib +export THRIFT_INC_DIR=$TOOLSDIR/thrift-0.9.0/include + fi + if [ -n "$HBASE_CNF_DIR" -a -n "$HADOOP_CNF_DIR" -a \ --- End diff -- Good point, I should not have removed it from bldenvchk (did that when I tried to eliminate Thrift altogether - unsuccessfully). I'll put the check back into bldenvchk, because the build would fail. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955201#comment-14955201 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41888212 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files --- End diff -- Yes, thanks. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955203#comment-14955203 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41888302 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} +tar -xzf ${PROTOBUF_TAR} >>${LOGFILE} + fi + + if [[ ! -d $PROTOBUF_TGT_ID ]]; then +cd ${PROTOBUF_ID} +echo "Building Google Protocol Buffers, this could take a while..." | tee -a ${LOGFILE} +if [[ $VERBOSE == true ]]; then --- End diff -- Why can't we just download binaries for protocol buffers? > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/exec
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955204#comment-14955204 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41888635 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + --- End diff -- Sorry, I don't remember anymore why I picked 2.6.0, but it does work. We use 2.5.0-cdh5.3.0 elsewhere, which is different from Apache 2.5.0, and I might have run into some interface issues with the Apache 2.5.0 version. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955207#comment-14955207 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user robertamarton commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41888772 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} +tar -xzf ${PROTOBUF_TAR} >>${LOGFILE} + fi + + if [[ ! -d $PROTOBUF_TGT_ID ]]; then +cd ${PROTOBUF_ID} +echo "Building Google Protocol Buffers, this could take a while..." | tee -a ${LOGFILE} +if [[ $VERBOSE == true ]]; then + ./configure --prefix=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} 2>&1 | tee -a ${LOGFILE} +else + ./configure --prefix=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} 2>&1 >>${LOGFILE} +fi +if [[ $? != 0 ]]; then + echo "Error during configure step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +make 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then + echo "Error during make step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +# skip the tests +# make check 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then + echo "Error during check step, exiting" | tee -a ${LOGFILE} + exit 1 +fi +make install 2>&1 >>${LOGFILE} +if [[ $? != 0 ]]; then + echo "Error during install step, exiting" | tee -a ${LOGFILE} + # remove partial results, if any + rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + exit 1 +fi + fi + + cd $LIBHDFS_TEMP_DIR + export HADOOP_PROTOC_PATH=${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID}/bin/protoc + + if [[ ! -f ${HADOOP_SRC_TAR} ]]; then +echo "Downloading Hadoop tar file ${HA
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955208#comment-14955208 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r4124 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + --- End diff -- It might also have been because at some point Apache started to distribute the 64 bit native library in the binary distribution - that would not be a strong reason of course, since we are building the library here anyway. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955223#comment-14955223 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41889662 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + --- End diff -- I left out libhadoop.so on purpose. Some of the tar files don't seem to include it and it is not needed to build, only at runtime. There might be a problem if it is missing both at compile time and at runtime. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955225#comment-14955225 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41889869 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi --- End diff -- Thanks, will fix that. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955232#comment-14955232 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41890253 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; + +*) + echo "Unknown command line option: $arg" + echo "Usage $0 [ -f | --force ]" + exit 1 +;; + esac + + shift +done + + +if [[ $FORCE_BUILD == true || \ + ! -e ${TGT_INC_DIR}/hdfs.h || \ + ! -e ${TGT_LIB_DIR}/libhdfs.so ]]; then + + if [ ! -d $LIBHDFS_TEMP_DIR ]; then +mkdir $LIBHDFS_TEMP_DIR + fi + + cd $LIBHDFS_TEMP_DIR + + if [[ ! -f ${PROTOBUF_TAR} ]]; then +echo "Downloading Google Protocol Buffers..." | tee -a ${LOGFILE} +wget ${PROTOBUF_MIRROR_URL}/${PROTOBUF_TAR} >${LOGFILE} + fi + + if [[ $FORCE_BUILD == true ]]; then +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_ID} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} + fi + + if [[ ! -d ${PROTOBUF_ID} ]]; then +echo "Unpacking Google Protocol Buffer tar file..." | tee -a ${LOGFILE} +rm -rf ${LIBHDFS_TEMP_DIR}/${PROTOBUF_TGT_ID} +tar -xzf ${PROTOBUF_TAR} >>${LOGFILE} + fi + + if [[ ! -d $PROTOBUF_TGT_ID ]]; then +cd ${PROTOBUF_ID} +echo "Building Google Protocol Buffers, this could take a while..." | tee -a ${LOGFILE} +if [[ $VERBOSE == true ]]; then --- End diff -- I didn't see a place where you can download binaries. The RPMs that are available for protoc on CentOS don't include version 2.5.0. Let me know if you find a good place to download this as a binary. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiri
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955242#comment-14955242 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user sandhyasun commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41891342 --- Diff: core/sqf/sqenvcom.sh --- @@ -580,6 +547,13 @@ EOF echo " ERROR: Unable to determine location of HBase lib directory" fi + if [[ -d $TOOLSDIR/thrift-0.9.0 ]]; then +# this is mostly for a build environment, where we need +# thrift from TOOLSDIR +export THRIFT_LIB_DIR=$TOOLSDIR/thrift-0.9.0/lib +export THRIFT_INC_DIR=$TOOLSDIR/thrift-0.9.0/include + fi + if [ -n "$HBASE_CNF_DIR" -a -n "$HADOOP_CNF_DIR" -a \ --- End diff -- Is CURL_INC_DIR and CURL_LIB_DIR removed ? We need it for LOB support. The code is not yet active due to a few issues hence no regression test yet for this case. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955249#comment-14955249 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41892116 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + --- End diff -- Yes, good idea, I'll add that. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955279#comment-14955279 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41894882 --- Diff: core/sqf/sqenvcom.sh --- @@ -580,6 +547,13 @@ EOF echo " ERROR: Unable to determine location of HBase lib directory" fi + if [[ -d $TOOLSDIR/thrift-0.9.0 ]]; then +# this is mostly for a build environment, where we need +# thrift from TOOLSDIR +export THRIFT_LIB_DIR=$TOOLSDIR/thrift-0.9.0/lib +export THRIFT_INC_DIR=$TOOLSDIR/thrift-0.9.0/include + fi + if [ -n "$HBASE_CNF_DIR" -a -n "$HADOOP_CNF_DIR" -a \ --- End diff -- No, these are still there. Good to know, I was wondering what those are used for. Thanks for checking. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955285#comment-14955285 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/118#discussion_r41895218 --- Diff: core/sqf/sql/scripts/get_libhdfs_files --- @@ -0,0 +1,186 @@ +#!/bin/sh + +# This script downloads and/or makes the required libhdfs files +# to be able to build Trafodion, which acts as a libhdfs client. +# +# Basically, what we need are three files: +# +# hdfs.h (copied to $TGT_INC_DIR) +# libhdfs.so (copied to $TGT_LIB_DIR) +# libhadoop.so (copied to $TGT_LIB_DIR) + +# Working dir in the Trafodion source tree to extract and build libhdfs files +# (can be specified as an environment variable) +if [[ -z ${LIBHDFS_TEMP_DIR} ]]; then + LIBHDFS_TEMP_DIR=${MY_SQROOT}/sql/libhdfs_files +fi +LOGFILE=${LIBHDFS_TEMP_DIR}/build.log + +# Hadoop source tar file to build libhdfs from +HADOOP_SRC_MIRROR_URL=https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0 +HADOOP_ID=hadoop-2.6.0 +HADOOP_SRC_ID=${HADOOP_ID}-src +HADOOP_SRC_TAR=${HADOOP_SRC_ID}.tar.gz + +# files to build required version of Google Protocol Buffers +PROTOBUF_MIRROR_URL=https://github.com/google/protobuf/releases/download/v2.5.0 +PROTOBUF_ID=protobuf-2.5.0 +PROTOBUF_TAR=${PROTOBUF_ID}.tar.gz + +# result of protobuf build +PROTOBUF_TGT_ID=protobuf-tgt + +# Directories to copy the built libhdfs library and corresponding include file +TGT_INC_DIR=$MY_SQROOT/export/include +TGT_LIB_DIR=$MY_SQROOT/export/lib${SQ_MBTYPE} + +FORCE_BUILD=false +VERBOSE=false + +USAGE="Usage $0 [ -f | --force ]\ +[ -v | --verbose ]\ +[ -d | --tempDir ]" + +while [[ $# > 0 ]] +do + arg="$1" + + case $arg in +-f|--force) + FORCE_BUILD=true + ;; + +-v|--verbose) + VERBOSE=true + ;; + +-d|--tempDir) + shift + if [[ $# < 1 ]]; then + echo "Expecting argument after -d or --tempDir" + exit 1 + fi + LIBHDFS_TEMP_DIR="$1" + ;; --- End diff -- Yes, the command line options should override environment variables, if specified (mixing the two is probably not a good idea). I will fix the log file issue and update the LOGFILE variable here. Thanks. > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1521) Trafodion build should not require HBase to be installed
[ https://issues.apache.org/jira/browse/TRAFODION-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956084#comment-14956084 ] ASF GitHub Bot commented on TRAFODION-1521: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/118 > Trafodion build should not require HBase to be installed > > > Key: TRAFODION-1521 > URL: https://issues.apache.org/jira/browse/TRAFODION-1521 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 1.1 (pre-incubation) > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Labels: build > Fix For: 1.2-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > When we build Trafodion in a source tree, it should not be necessary to > install HBase first. > Until now, several build steps relied on an HBase installation: > 1. Files in core/sql/executor include the libhdfs header file, hdfs.h. This > file isn't available in the Trafodion source tree. > 2. When we link certain executables like mxosrvr and sqlci, we specify > -lhdfs, requiring the libhdfs.so DLL as well as the DLLs it depends on, > libhadoop.so and libjvm.so. > 3. Until recently, executor Java (core/sql/executor) files were compiled with > javac invocations from a Makefile, requiring the HBase client jar files in > the CLASSPATH. This got changed with TRAFODION-1502. > 4. I missed some more Java files in the fix for TRAFODION-1502: > core/sqf/src/seatrans/tm/hbasetmlib2. Those should also be moved to Maven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)