Hi I am trying to write a spark code that reads data from Hbase and store it in DataFrame. I am able to run it perfectly with hbase-site.xml in $spark-home/conf folder. But I am facing few issues Here.
Issue 1: Passing hbase-site.xml location with --file parameter submitted through client mode(It is working in cluster mode) When I removed hbase-site.xml from spark/conf and try to execute it in the client mode by passing with file --file parameter over yarn I keep getting the following exception. Which I think it means it is not taking the zookeeper configuration from hbase-site.xml. How ever it works good when i run it in cluster mode. sample command spark-submit --master yarn --deploy-mode cluster --files /home/siddesh/hbase-site.xml --class com.orzota.rs.json.HbaseConnector --packages com.hortonworks:shc:1.0.0-2.0-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ target/scala-2.11/test-0.1-SNAPSHOT.jar at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 18/02/22 01:43:09 INFO ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) 18/02/22 01:43:09 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) Issue 2: Passing hbase configuration details through spark session(Not working in cluster as well as client mode). Instead of passing the entire hbase-site.xml I am trying to add the configuration directly in the spark code by adding it as a config parameter in spark session the following is a sample spark-session command. val spark = SparkSession .builder() .appName(name) .config("hbase.zookeeper.property.clientPort", "2181") .config("hbase.zookeeper.quorum", "ip1,ip2,ip3") .config("spark.hbase.host","zookeeperquorum") .getOrCreate() val json_df = spark.read.option("catalog",catalog_read).format("org.apache.spark.sql.execution.datasources.hbase").load() But it is not working in the cluster mode while the issue-1 continues in the client mode. Can anyone help me with a solution or explanation why this is happening are there any work arounds ??. regards Sid
