Hi All,
I have configured Spark to query on hive table.
Run the Thrift JDBC/ODBC server using below command :
*cd $SPARK_HOME*
*./sbin/start-thriftserver.sh --master spark://myhost:7077 --hiveconf
hive.server2.thrift.bind.host=myhost --hiveconf
hive.server2.thrift.port=9999*
and also able to connect through beeline
*beeline>* !connect jdbc:hive2://192.168.145.20:9999
Enter username for jdbc:hive2://192.168.145.20:9999: root
Enter password for jdbc:hive2://192.168.145.20:9999: impetus
*beeline > *
It is not giving query result on hive table through Spark JDBC, but it is
working with spark HiveSQLContext. See complete scenario explain below.
Help me understand the issue why Spark SQL JDBC is not giving result ?
Below are version details.
*Hive Version : 1.2.1*
*Hadoop Version : 2.6.0*
*Spark version : 1.3.1*
Let me know if need other details.
*Created Hive Table , insert some records and query it :*
*beeline> !connect jdbc:hive2://myhost:10000*
Enter username for jdbc:hive2://myhost:10000: root
Enter password for jdbc:hive2://myhost:10000: ******
*beeline> create table tampTable(id int ,name string ) clustered by (id)
into 2 buckets stored as orc TBLPROPERTIES('transactional'='true');*
*beeline> insert into table tampTable values
(1,'row1'),(2,'row2'),(3,'row3');*
*beeline> select name from tampTable;*
name
---------
row1
row3
row2
*Query through SparkSQL HiveSQLContext :*
SparkConf sparkConf = new SparkConf().setAppName("JavaSparkSQL");
SparkContext sc = new SparkContext(sparkConf);
HiveContext hiveContext = new HiveContext(sc);
DataFrame teenagers = hiveContext.sql("*SELECT name FROM tampTable*");
List<String> teenagerNames = teenagers.toJavaRDD().map(new Function<Row,
String>() {
@Override
public String call(Row row) {
return "Name: " + row.getString(0);
}
}).collect();
for (String name: teenagerNames) {
System.out.println(name);
}
teenagers2.toJavaRDD().saveAsTextFile("/tmp1");
sc.stop();
which is working perfectly and giving all names from table *tempTable*
*Query through Spark SQL JDBC :*
*beeline> !connect jdbc:hive2://myhost:9999*
Enter username for jdbc:hive2://myhost:9999: root
Enter password for jdbc:hive2://myhost:9999: ******
*beeline> show tables;*
*temptable*
*..other tables*
beeline> *SELECT name FROM tampTable;*
I can list the table through "show tables", but I run the query , it is
either hanged or returns nothing.
Regards
Sanjiv Singh
Mob : +091 9990-447-339
