Hi,
I am getting error when I am trying to connect hive table (which is being
created through HbaseIntegration) in spark
Steps I followed :
*Hive Table creation code *:
CREATE EXTERNAL TABLE IF NOT EXISTS TEST(NAME STRING,AGE INT)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,0:AGE")
TBLPROPERTIES ("hbase.table.name" = "TEST",
"hbase.mapred.output.outputtable" = "TEST");
*DESCRIBE TEST ;*
col_name data_type comment
name string from deserializer
age int from deserializer
*Spark Code :*
import org.apache.spark._
import org.apache.spark.sql._
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
hiveContext.sql("from TEST SELECT NAME").collect.foreach(println)
*Starting Spark shell*
spark-shell --jars
/usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar
--driver-class-path
/usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar
--packages com.databricks:spark-csv_2.10:1.3.0 --master yarn-client -i
/TestDivya/Spark/InstrumentCopyToHDFSHive.scala
*Stack Trace* :
Stack SQL context available as sqlContext.
> Loading /TestDivya/Spark/InstrumentCopyToHDFSHive.scala...
> import org.apache.spark._
> import org.apache.spark.sql._
> 16/02/29 23:09:29 INFO HiveContext: Initializing execution hive, version
> 1.2.1
> 16/02/29 23:09:29 INFO ClientWrapper: Inspected Hadoop version:
> 2.7.1.2.3.4.0-3485
> 16/02/29 23:09:29 INFO ClientWrapper: Loaded
> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version
> 2.7.1.2.3.4.0-3485
> 16/02/29 23:09:29 INFO HiveContext: default warehouse location is
> /user/hive/warehouse
> 16/02/29 23:09:29 INFO HiveContext: Initializing HiveMetastoreConnection
> version 1.2.1 using Spark classes.
> 16/02/29 23:09:29 INFO ClientWrapper: Inspected Hadoop version:
> 2.7.1.2.3.4.0-3485
> 16/02/29 23:09:29 INFO ClientWrapper: Loaded
> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version
> 2.7.1.2.3.4.0-3485
> 16/02/29 23:09:30 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/02/29 23:09:30 INFO metastore: Trying to connect to metastore with URI
> thrift://ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal:9083
> 16/02/29 23:09:30 INFO metastore: Connected to metastore.
> 16/02/29 23:09:30 WARN DomainSocketFactory: The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.
> 16/02/29 23:09:31 INFO SessionState: Created local directory:
> /tmp/1bf53785-f7c8-406d-a733-a5858ccb2d16_resources
> 16/02/29 23:09:31 INFO SessionState: Created HDFS directory:
> /tmp/hive/hdfs/1bf53785-f7c8-406d-a733-a5858ccb2d16
> 16/02/29 23:09:31 INFO SessionState: Created local directory:
> /tmp/hdfs/1bf53785-f7c8-406d-a733-a5858ccb2d16
> 16/02/29 23:09:31 INFO SessionState: Created HDFS directory:
> /tmp/hive/hdfs/1bf53785-f7c8-406d-a733-a5858ccb2d16/_tmp_space.db
> hiveContext: org.apache.spark.sql.hive.HiveContext =
> org.apache.spark.sql.hive.HiveContext@10b14f32
> 16/02/29 23:09:32 INFO ParseDriver: Parsing command: from TEST SELECT NAME
> 16/02/29 23:09:32 INFO ParseDriver: Parse Completed
> 16/02/29 23:09:33 INFO deprecation: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 16/02/29 23:09:33 INFO MemoryStore: ensureFreeSpace(468352) called with
> curMem=0, maxMem=556038881
> 16/02/29 23:09:33 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 457.4 KB, free 529.8 MB)
> 16/02/29 23:09:33 INFO MemoryStore: ensureFreeSpace(49454) called with
> curMem=468352, maxMem=556038881
> 16/02/29 23:09:33 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 48.3 KB, free 529.8 MB)
> 16/02/29 23:09:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on xxx.xx.xx.xxx:37784 (size: 48.3 KB, free: 530.2 MB)
> 16/02/29 23:09:33 INFO SparkContext: Created broadcast 0 from collect at
> <console>:30
> 16/02/29 23:09:34 INFO HBaseStorageHandler: Configuring input job
> properties
> 16/02/29 23:09:34 INFO RecoverableZooKeeper: Process
> identifier=hconnection-0x26fa89a2 connecting to ZooKeeper
> ensemble=localhost:2181
> 16/02/29 23:09:34 INFO ZooKeeper: Client
> environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:host.name
> =ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.version=1.7.0_67
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.vendor=Oracle
> Corporation
> 16/02/29 23:09:34 INFO ZooKeeper: Client
> environment:java.home=/usr/jdk64/jdk1.7.0_67/jre
> 16/02/29 23:09:34 INFO ZooKeeper: Client
> environment:java.class.path=/usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar:/usr/hdp/current/spark-thriftserver/conf/:/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-core-3.2.10.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/
> 16/02/29 23:09:34 INFO ZooKeeper: Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.compiler=<NA>
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:os.name=Linux
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:os.arch=amd64
> 16/02/29 23:09:34 INFO ZooKeeper: Client
> environment:os.version=3.10.0-229.el7.x86_64
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:user.name=hdfs
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:user.home=/home/hdfs
> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:user.dir=/home/hdfs
> 16/02/29 23:09:34 INFO ZooKeeper: Initiating client connection,
> connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x26fa89a20x0, quorum=localhost:2181, baseZNode=/hbase
> 16/02/29 23:09:34 INFO ClientCnxn: Opening socket connection to server
> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 16/02/29 23:09:34 INFO ClientCnxn: Socket connection established to
> localhost/0:0:0:0:0:0:0:1:2181, initiating session
> 16/02/29 23:09:34 INFO ClientCnxn: Session establishment complete on
> server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3532fb70ba20034,
> negotiated timeout = 40000
> 16/02/29 23:09:34 WARN TableInputFormatBase: You are using an HTable
> instance that relies on an HBase-managed Connection. This is usually due to
> directly creating an HTable, which is deprecated. Instead, you should
> create a Connection object and then request a Table instance from it. If
> you don't need the Table instance for your own use, you should instead use
> the TableInputFormatBase.initalizeTable method directly.
> 16/02/29 23:09:34 INFO TableInputFormatBase: Creating an additional
> unmanaged connection because user provided one can't be used for
> administrative actions. We'll close it when we close out the table.
> 16/02/29 23:09:34 INFO RecoverableZooKeeper: Process
> identifier=hconnection-0x6fd74d35 connecting to ZooKeeper
> ensemble=localhost:2181
> 16/02/29 23:09:34 INFO ZooKeeper: Initiating client connection,
> connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x6fd74d350x0, quorum=localhost:2181, baseZNode=/hbase
> 16/02/29 23:09:34 INFO ClientCnxn: Opening socket connection to server
> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 16/02/29 23:09:34 INFO ClientCnxn: Socket connection established to
> localhost/0:0:0:0:0:0:0:1:2181, initiating session
> 16/02/29 23:09:34 INFO ClientCnxn: Session establishment complete on
> server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3532fb70ba20035,
> negotiated timeout = 40000
> 16/02/29 23:09:34 INFO RegionSizeCalculator: Calculating region sizes for
> table "TEST".
> 16/02/29 23:10:23 INFO RpcRetryingCaller: Call exception, tries=10,
> retries=35, started=48318 ms ago, cancelled=false, msg=
> 16/02/29 23:10:43 INFO RpcRetryingCaller: Call exception, tries=11,
> retries=35, started=68524 ms ago, cancelled=false, msg=
> 16/02/29 23:11:03 INFO RpcRetryingCaller: Call exception, tries=12,
> retries=35, started=88617 ms ago, cancelled=false, msg=
> 16/02/29 23:11:23 INFO RpcRetryingCaller: Call exception, tries=13,
> retries=35, started=108676 ms ago, cancelled=false, msg=
> 16/02/29 23:11:43 INFO RpcRetryingCaller: Call exception, tries=14,
> retries=35, started=128747 ms ago, cancelled=false, msg=
> 16/02/29 23:12:03 INFO RpcRetryingCaller: Call exception, tries=15,
> retries=35, started=148938 ms ago, cancelled=false, msg=
> 16/02/29 23:12:23 INFO RpcRetryingCaller: Call exception, tries=16,
> retries=35, started=168942 ms ago, cancelled=false, msg=
> 16/02/29 23:12:43 INFO RpcRetryingCaller: Call exception, tries=17,
> retries=35, started=188975 ms ago, cancelled=false, msg=
> Trace :
Could somebody help me in resolving the error.
Would really appreciate the help .
Thanks,
Divya
