[
https://issues.apache.org/jira/browse/KYLIN-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411835#comment-17411835
]
曹勇 commented on KYLIN-5085:
---------------------------
[~xiacongling]
{code:java}
[hadoop@node2 ~]$ hive -e set
which: no hbase in
(/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/zookeeper-3.7.0/bin:/usr/local/firefox:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre/bin:/home/hadoop/hadoop-3.2.2/bin:/home/hadoop/hadoop-3.2.2/sbin/:/home/hadoop/spark-3.1.2-bin-hadoop3.2/bin:/home/hadoop/spark-3.1.2-bin-hadoop3.2/sbin:/home/hadoop/hive-2.3.9/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hadoop/hive-2.3.9/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/hadoop/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:4051)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:4009)
at
org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:81)
at
org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:65)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:702)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
{code}
Then I replace guava version for hive:
{code:java}
[hadoop@node2 ~]$ cp $HADOOP_HOME/share/hadoop/common/lib/guava-27.0-jre.jar
$HIVE_HOME/lib
[hadoop@node2 ~]$ rm -rf $HIVE_HOME/lib/guava-14.0.1.jar
[hadoop@node2 ~]$ rm -rf
$KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib/guava-14.0.jar
[hadoop@node2 ~]$ cp $HADOOP_HOME/share/hadoop/common/lib/guava-27.0-jre.jar
$KYLIN_HOME/ext/
{code}
And the command `hive -e set` is OK.
And kylin daemon server will be OK.
thx.
> Something wrong with Hive CLI or Beeline, please execute Hive CLI or Beeline
> CLI in terminal to find the root cause.
> --------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-5085
> URL: https://issues.apache.org/jira/browse/KYLIN-5085
> Project: Kylin
> Issue Type: Bug
> Components: Client - CLI, Environment
> Affects Versions: v4.0.0
> Environment: hive-2.3.9
> kylin-4.0.0
> spark-3.1.2-bin-hadoop3.2
> zookeeper-3.7.0
> Reporter: 曹勇
> Priority: Major
> Attachments: image-2021-09-07-20-40-15-503.png
>
>
> when I reduce hadoop version to hadoop-3.2.2 , run ./kylin.sh start
> the command export :
>
> {code:java}
> [hadoop@node3 logs]$ ../bin/kylin.sh start
> Retrieving hadoop conf dir...
> ...................................................[PASS]
> KYLIN_HOME is set to /home/hadoop/kylin-4.0.0
> Checking hive
> ...................................................[PASS]
> Checking hadoop shell
> ...................................................[PASS]
> Checking hdfs working dir
> ...................................................[PASS]Checking environment
> finished successfully. To check again, run 'bin/check-env.sh' manually.
> Retrieving hive dependency...
> Something wrong with Hive CLI or Beeline, please execute Hive CLI or Beeline
> CLI in terminal to find the root cause.
> {code}
> Then I set `set -exv` in the $KYLIN_HOME/bin/find-hive-dependency.sh
> !image-2021-09-07-20-40-15-503.png!
> And the script print:
> {code:java}
> [hadoop@node3 bin]$ ./kylin.sh start
> Retrieving hadoop conf dir...
> ...................................................[PASS]
> KYLIN_HOME is set to /home/hadoop/kylin-4.0.0
> Checking hive
> ...................................................[PASS]
> Checking hadoop shell
> ...................................................[PASS]
> Checking hdfs working dir
> ...................................................[PASS]Checking environment
> finished successfully. To check again, run 'bin/check-env.sh' manually.
> source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
> -P)/../"}/bin/header.sh
> ++ source /home/hadoop/kylin-4.0.0/bin/header.sh
> #!/bin/bash#
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> ## source meverbose=${verbose:-""}
> +++ verbose=while getopts ":v" opt; do
> case $opt in
> v)
> echo "Turn on verbose mode." >&2
> verbose=true
> ;;
> \?)
> echo "Invalid option: -$OPTARG" >&2
> ;;
> esac
> done
> +++ getopts :v optif [[ "$dir" == "" ]]
> then
> dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set
> KYLIN_HOME with consideration for multiple instances that are on the same node
> KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"}
> export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd`
> dir="$KYLIN_HOME/bin" function quit {
> echo "$@"
> exit 1
> } function verbose {
> if [[ -n "$verbose" ]]; then
> echo "$@"
> fi
> } function setColor() {
> echo -e "\033[$1m$2\033[0m"
> } # set JAVA
> if [[ "${JAVA}" == "" ]]; then
> if [[ -z "$JAVA_HOME" ]]; then
> JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print
> $2}'`
> if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then
> JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
> else
> quit "Java 1.8 or above is required."
> fi
> [[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME"
> export JAVA_HOME
> fi
> export JAVA=$JAVA_HOME/bin/java
> [[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set
> JAVA_HOME correctly."
> verbose "java is ${JAVA}"
> fi
> fi
> +++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]## ${dir} assigned to
> $KYLIN_HOME/bin in header.sh
> source ${dir}/load-hive-conf.sh
> ++ source /home/hadoop/kylin-4.0.0/bin/load-hive-conf.sh
> #!/bin/bash#
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
> -P)/../"}/bin/header.sh
> +++ source /home/hadoop/kylin-4.0.0/bin/header.sh
> #!/bin/bash#
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> ## source meverbose=${verbose:-""}
> ++++ verbose=while getopts ":v" opt; do
> case $opt in
> v)
> echo "Turn on verbose mode." >&2
> verbose=true
> ;;
> \?)
> echo "Invalid option: -$OPTARG" >&2
> ;;
> esac
> done
> ++++ getopts :v optif [[ "$dir" == "" ]]
> then
> dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set
> KYLIN_HOME with consideration for multiple instances that are on the same node
> KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"}
> export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd`
> dir="$KYLIN_HOME/bin" function quit {
> echo "$@"
> exit 1
> } function verbose {
> if [[ -n "$verbose" ]]; then
> echo "$@"
> fi
> } function setColor() {
> echo -e "\033[$1m$2\033[0m"
> } # set JAVA
> if [[ "${JAVA}" == "" ]]; then
> if [[ -z "$JAVA_HOME" ]]; then
> JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print
> $2}'`
> if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then
> JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
> else
> quit "Java 1.8 or above is required."
> fi
> [[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME"
> export JAVA_HOME
> fi
> export JAVA=$JAVA_HOME/bin/java
> [[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set
> JAVA_HOME correctly."
> verbose "java is ${JAVA}"
> fi
> fi
> ++++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]# source me
> hive_conf_dir="${KYLIN_HOME}/conf/kylin_hive_conf.xml"
> +++
> hive_conf_dir=/home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xmlnames=(`sed -n
> '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<name>\(.*\)</name>|\1|p' | sed
> 's/ \+//g'`)
> +++ names=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
> 's|<name>\(.*\)</name>|\1|p' | sed 's/ \+//g'`)
> sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
> 's|<name>\(.*\)</name>|\1|p' | sed 's/ \+//g'
> ++++ sed -n 's|<name>\(.*\)</name>|\1|p'
> ++++ sed -n '/<!--/,/-->/!p' /home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xml
> ++++ sed 's/ \+//g'
> values=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
> 's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'`)
> +++ values=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
> 's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'`)
> sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
> 's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'
> ++++ sed -n 's|<value>\(.*\)</value>|\1|p'
> ++++ sed -n '/<!--/,/-->/!p' /home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xml
> ++++ sed 's/ \+//g'len_names=${#names[@]}
> +++ len_names=9
> len_values=${#values[@]}
> +++ len_values=9[[ $len_names == $len_values ]] || quit "Current hive conf
> file: ${hive_conf_dir} has inconsistent key value pairs."
> +++ [[ 9 == 9 ]]for((i=0;i<$len_names;i++))
> do
> hive_conf_properties=${hive_conf_properties}" --hiveconf
> ${names[$i]}=${values[$i]} "
> done
> +++ (( i=0 ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
> hive.auto.convert.join.noconditionaltask.size=100000000 '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
> hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
> hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf
> hive.stats.autogather=true '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
> hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf
> hive.stats.autogather=true --hiveconf hive.merge.mapfiles=false '
> +++ (( i++ ))
> +++ (( i<9 ))
> +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
> hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf
> hive.stats.autogather=true --hiveconf hive.merge.mapfiles=false --hiveconf
> hive.merge.mapredfiles=false '
> +++ (( i++ ))
> +++ (( i<9 ))echo Retrieving hive dependency...
> ++ echo Retrieving hive dependency...
> Retrieving hive dependency...client_mode=`bash
> ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
> bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client
> +++ bash /home/hadoop/kylin-4.0.0/bin/get-properties.sh
> kylin.source.hive.client
> ++ client_mode=
> hive_env=
> ++ hive_env=if [ "${client_mode}" == "beeline" ]
> then
> beeline_shell=`$KYLIN_HOME/bin/get-properties.sh
> kylin.source.hive.beeline-shell`
> beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh
> kylin.source.hive.beeline-params`
> hive_env=`${beeline_shell} ${hive_conf_properties} ${beeline_params}
> --outputformat=dsv -e "set;" 2>&1 | grep --text 'env:CLASSPATH' `
> else
> source ${dir}/check-hive-usability.sh
> hive_env=`hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'`
> fi
> ++ '[' '' == beeline ']'
> ++ source /home/hadoop/kylin-4.0.0/bin/check-hive-usability.sh
> #!/bin/bash#
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
> -P)/../"}/bin/header.sh
> +++ source /home/hadoop/kylin-4.0.0/bin/header.sh
> #!/bin/bash#
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> ## source meverbose=${verbose:-""}
> ++++ verbose=while getopts ":v" opt; do
> case $opt in
> v)
> echo "Turn on verbose mode." >&2
> verbose=true
> ;;
> \?)
> echo "Invalid option: -$OPTARG" >&2
> ;;
> esac
> done
> ++++ getopts :v optif [[ "$dir" == "" ]]
> then
> dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set
> KYLIN_HOME with consideration for multiple instances that are on the same node
> KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"}
> export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd`
> dir="$KYLIN_HOME/bin" function quit {
> echo "$@"
> exit 1
> } function verbose {
> if [[ -n "$verbose" ]]; then
> echo "$@"
> fi
> } function setColor() {
> echo -e "\033[$1m$2\033[0m"
> } # set JAVA
> if [[ "${JAVA}" == "" ]]; then
> if [[ -z "$JAVA_HOME" ]]; then
> JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print
> $2}'`
> if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then
> JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
> else
> quit "Java 1.8 or above is required."
> fi
> [[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME"
> export JAVA_HOME
> fi
> export JAVA=$JAVA_HOME/bin/java
> [[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set
> JAVA_HOME correctly."
> verbose "java is ${JAVA}"
> fi
> fi
> ++++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]hive -e "select 1" > /dev/null
> 2>&1 &
> pid=$!
> +++ pid=25366((timeLeft = 60))
> +++ (( timeLeft = 60 ))while ((timeLeft > 0)); do
> sleep 5
> ps -p $pid > /dev/null || break
> ((timeLeft -= 5))
> done
> +++ hive -e 'select 1'
> +++ (( timeLeft > 0 ))
> +++ sleep 5
> +++ ps -p 25366
> +++ breakif ps -p $pid > /dev/null
> then
> kill $pid
> sleep 5 # still alive, use kill -9
> if ps -p $pid > /dev/null
> then
> kill -9 $pid
> fi quit "ERROR: Check hive's usability failed, please check the status
> of your cluster"
> fi
> +++ ps -p 25366
> hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'
> +++ hive --hiveconf dfs.replication=2 --hiveconf
> hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
> --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
> hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true
> --hiveconf hive.merge.mapfiles=false --hiveconf hive.merge.mapredfiles=false
> -e set
> +++ grep env:CLASSPATH
> ++ hive_env=
> {code}
> I can't really tell what the problem is .
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
