Hello Divya! Have you solved the problem? I suppose the log comes from driver. You need to look also at logs on worker JVMs, there can be an exception or something. Do you have Kerberos on your cluster? It could be similar to a problem http://issues.apache.org/jira/browse/SPARK-14115
Based on your logs: > 16/02/29 23:09:34 INFO ClientCnxn: Opening socket connection to server > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL > (unknown error) > 16/02/29 23:09:34 INFO ClientCnxn: Socket connection established to > localhost/0:0:0:0:0:0:0:1:2181, initiating session > 16/02/29 23:09:34 INFO ClientCnxn: Session establishment complete on > server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3532fb70ba20035, Maybe there is a problem with using RPC call to regions using IPv6 (but I just guess). -- Kind regards/ Pozdrawiam, Wojciech Indyk http://datacentric.pl 2016-03-01 5:27 GMT+01:00 Divya Gehlot <divya.htco...@gmail.com>: > Hi, > I am getting error when I am trying to connect hive table (which is being > created through HbaseIntegration) in spark > > Steps I followed : > *Hive Table creation code *: > CREATE EXTERNAL TABLE IF NOT EXISTS TEST(NAME STRING,AGE INT) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,0:AGE") > TBLPROPERTIES ("hbase.table.name" = "TEST", > "hbase.mapred.output.outputtable" = "TEST"); > > > *DESCRIBE TEST ;* > col_name data_type comment > name string from deserializer > age int from deserializer > > > *Spark Code :* > import org.apache.spark._ > import org.apache.spark.sql._ > > val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > hiveContext.sql("from TEST SELECT NAME").collect.foreach(println) > > > *Starting Spark shell* > spark-shell --jars > /usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar > --driver-class-path > /usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar > --packages com.databricks:spark-csv_2.10:1.3.0 --master yarn-client -i > /TestDivya/Spark/InstrumentCopyToHDFSHive.scala > > *Stack Trace* : > > Stack SQL context available as sqlContext. >> Loading /TestDivya/Spark/InstrumentCopyToHDFSHive.scala... >> import org.apache.spark._ >> import org.apache.spark.sql._ >> 16/02/29 23:09:29 INFO HiveContext: Initializing execution hive, version >> 1.2.1 >> 16/02/29 23:09:29 INFO ClientWrapper: Inspected Hadoop version: >> 2.7.1.2.3.4.0-3485 >> 16/02/29 23:09:29 INFO ClientWrapper: Loaded >> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version >> 2.7.1.2.3.4.0-3485 >> 16/02/29 23:09:29 INFO HiveContext: default warehouse location is >> /user/hive/warehouse >> 16/02/29 23:09:29 INFO HiveContext: Initializing HiveMetastoreConnection >> version 1.2.1 using Spark classes. >> 16/02/29 23:09:29 INFO ClientWrapper: Inspected Hadoop version: >> 2.7.1.2.3.4.0-3485 >> 16/02/29 23:09:29 INFO ClientWrapper: Loaded >> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version >> 2.7.1.2.3.4.0-3485 >> 16/02/29 23:09:30 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> 16/02/29 23:09:30 INFO metastore: Trying to connect to metastore with URI >> thrift://ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal:9083 >> 16/02/29 23:09:30 INFO metastore: Connected to metastore. >> 16/02/29 23:09:30 WARN DomainSocketFactory: The short-circuit local reads >> feature cannot be used because libhadoop cannot be loaded. >> 16/02/29 23:09:31 INFO SessionState: Created local directory: >> /tmp/1bf53785-f7c8-406d-a733-a5858ccb2d16_resources >> 16/02/29 23:09:31 INFO SessionState: Created HDFS directory: >> /tmp/hive/hdfs/1bf53785-f7c8-406d-a733-a5858ccb2d16 >> 16/02/29 23:09:31 INFO SessionState: Created local directory: >> /tmp/hdfs/1bf53785-f7c8-406d-a733-a5858ccb2d16 >> 16/02/29 23:09:31 INFO SessionState: Created HDFS directory: >> /tmp/hive/hdfs/1bf53785-f7c8-406d-a733-a5858ccb2d16/_tmp_space.db >> hiveContext: org.apache.spark.sql.hive.HiveContext = >> org.apache.spark.sql.hive.HiveContext@10b14f32 >> 16/02/29 23:09:32 INFO ParseDriver: Parsing command: from TEST SELECT NAME >> 16/02/29 23:09:32 INFO ParseDriver: Parse Completed >> 16/02/29 23:09:33 INFO deprecation: mapred.map.tasks is deprecated. >> Instead, use mapreduce.job.maps >> 16/02/29 23:09:33 INFO MemoryStore: ensureFreeSpace(468352) called with >> curMem=0, maxMem=556038881 >> 16/02/29 23:09:33 INFO MemoryStore: Block broadcast_0 stored as values in >> memory (estimated size 457.4 KB, free 529.8 MB) >> 16/02/29 23:09:33 INFO MemoryStore: ensureFreeSpace(49454) called with >> curMem=468352, maxMem=556038881 >> 16/02/29 23:09:33 INFO MemoryStore: Block broadcast_0_piece0 stored as >> bytes in memory (estimated size 48.3 KB, free 529.8 MB) >> 16/02/29 23:09:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in >> memory on xxx.xx.xx.xxx:37784 (size: 48.3 KB, free: 530.2 MB) >> 16/02/29 23:09:33 INFO SparkContext: Created broadcast 0 from collect at >> <console>:30 >> 16/02/29 23:09:34 INFO HBaseStorageHandler: Configuring input job >> properties >> 16/02/29 23:09:34 INFO RecoverableZooKeeper: Process >> identifier=hconnection-0x26fa89a2 connecting to ZooKeeper >> ensemble=localhost:2181 >> 16/02/29 23:09:34 INFO ZooKeeper: Client >> environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:host.name >> =ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.version=1.7.0_67 >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.vendor=Oracle >> Corporation >> 16/02/29 23:09:34 INFO ZooKeeper: Client >> environment:java.home=/usr/jdk64/jdk1.7.0_67/jre >> 16/02/29 23:09:34 INFO ZooKeeper: Client >> environment:java.class.path=/usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server.jar:/usr/hdp/current/spark-thriftserver/conf/:/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-core-3.2.10.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/ >> 16/02/29 23:09:34 INFO ZooKeeper: Client >> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:java.compiler=<NA> >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:os.name=Linux >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:os.arch=amd64 >> 16/02/29 23:09:34 INFO ZooKeeper: Client >> environment:os.version=3.10.0-229.el7.x86_64 >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:user.name=hdfs >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:user.home=/home/hdfs >> 16/02/29 23:09:34 INFO ZooKeeper: Client environment:user.dir=/home/hdfs >> 16/02/29 23:09:34 INFO ZooKeeper: Initiating client connection, >> connectString=localhost:2181 sessionTimeout=90000 >> watcher=hconnection-0x26fa89a20x0, quorum=localhost:2181, baseZNode=/hbase >> 16/02/29 23:09:34 INFO ClientCnxn: Opening socket connection to server >> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL >> (unknown error) >> 16/02/29 23:09:34 INFO ClientCnxn: Socket connection established to >> localhost/0:0:0:0:0:0:0:1:2181, initiating session >> 16/02/29 23:09:34 INFO ClientCnxn: Session establishment complete on >> server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3532fb70ba20034, >> negotiated timeout = 40000 >> 16/02/29 23:09:34 WARN TableInputFormatBase: You are using an HTable >> instance that relies on an HBase-managed Connection. This is usually due to >> directly creating an HTable, which is deprecated. Instead, you should >> create a Connection object and then request a Table instance from it. If >> you don't need the Table instance for your own use, you should instead use >> the TableInputFormatBase.initalizeTable method directly. >> 16/02/29 23:09:34 INFO TableInputFormatBase: Creating an additional >> unmanaged connection because user provided one can't be used for >> administrative actions. We'll close it when we close out the table. >> 16/02/29 23:09:34 INFO RecoverableZooKeeper: Process >> identifier=hconnection-0x6fd74d35 connecting to ZooKeeper >> ensemble=localhost:2181 >> 16/02/29 23:09:34 INFO ZooKeeper: Initiating client connection, >> connectString=localhost:2181 sessionTimeout=90000 >> watcher=hconnection-0x6fd74d350x0, quorum=localhost:2181, baseZNode=/hbase >> 16/02/29 23:09:34 INFO ClientCnxn: Opening socket connection to server >> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL >> (unknown error) >> 16/02/29 23:09:34 INFO ClientCnxn: Socket connection established to >> localhost/0:0:0:0:0:0:0:1:2181, initiating session >> 16/02/29 23:09:34 INFO ClientCnxn: Session establishment complete on >> server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3532fb70ba20035, >> negotiated timeout = 40000 >> 16/02/29 23:09:34 INFO RegionSizeCalculator: Calculating region sizes for >> table "TEST". >> 16/02/29 23:10:23 INFO RpcRetryingCaller: Call exception, tries=10, >> retries=35, started=48318 ms ago, cancelled=false, msg= >> 16/02/29 23:10:43 INFO RpcRetryingCaller: Call exception, tries=11, >> retries=35, started=68524 ms ago, cancelled=false, msg= >> 16/02/29 23:11:03 INFO RpcRetryingCaller: Call exception, tries=12, >> retries=35, started=88617 ms ago, cancelled=false, msg= >> 16/02/29 23:11:23 INFO RpcRetryingCaller: Call exception, tries=13, >> retries=35, started=108676 ms ago, cancelled=false, msg= >> 16/02/29 23:11:43 INFO RpcRetryingCaller: Call exception, tries=14, >> retries=35, started=128747 ms ago, cancelled=false, msg= >> 16/02/29 23:12:03 INFO RpcRetryingCaller: Call exception, tries=15, >> retries=35, started=148938 ms ago, cancelled=false, msg= >> 16/02/29 23:12:23 INFO RpcRetryingCaller: Call exception, tries=16, >> retries=35, started=168942 ms ago, cancelled=false, msg= >> 16/02/29 23:12:43 INFO RpcRetryingCaller: Call exception, tries=17, >> retries=35, started=188975 ms ago, cancelled=false, msg= >> Trace : > > > > Could somebody help me in resolving the error. > Would really appreciate the help . > > > Thanks, > Divya --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
