Hello All, In Apache Phoenix homepage, It shows two additional functions: Apache Spark Integration and Phoenix Storage Handler for Apache Hive, According the guidance, I can query phoenix table from beeline-cli, I can load phoenix table as dataframe using Spark-sql. So my question is :
Does Phoenix support spark-sql query the hive external table mapped from Phoenix ? I am working on hdp3.0 ( Phoenix 5.0 Hbase 2.0, Hive 3.1.0 ,Spark2.3.1 ) and facing the issue as subject mentioned. I tried to solve this problem but failed, I found some similar questions on internet but the answers didn’t work for me. My submit command : spark-submit test3.py --jars \ /usr/hdp/current/phoenix-client/lib/phoenix-hive-5.0.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/phoenix-client/lib/hadoop-mapreduce-client-core.jar\ ,/usr/hdp/current/phoenix-client/lib/phoenix-core-5.0.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/phoenix-client/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/hive-client/lib/hive-metastore-3.1.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/hive-client/lib/hive-common-3.1.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/hive-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/hive-client/lib/hbase-mapreduce-2.0.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/hive-client/lib/hive-serde-3.1.0.3.0.0.0-1634.jar\ ,/usr/hdp/current/hive-client/lib/hive-shims-3.1.0.3.0.0.0-1634.jar Log attached and Demo code as below: from pyspark.sql import SparkSession if __name__ == '__main__': spark = SparkSession.builder \ .appName("test") \ .enableHiveSupport() \ .getOrCreate() df= spark.sql("select count(*) from ajmide_dw.part_device") df.show() Similar Issues: https://community.hortonworks.com/questions/140097/facing-issue-from-spark-sql.html https://stackoverflow.com/questions/51501044/unable-to-access-hive-external-tables-from-spark-shell Any comment or suggestion is appreciated! Thanks, Shi-Cheng, Ma