Are you certain you are providing Spark with the right Hive configuration?
Is there a valid HIVE_CONF_DIR defined in your spark-env.sh, with a
hive-site.xml detailing the location/etc. of the metastore service and/or
DB?
Without a valid metastore config, Hive may switch to using a local
(embedded) Derby DB and create metadata under that - thereby giving the
impression of things working, but not actually updating your true metastore
database.
You should be able to see your spark driver connect to your metastore if it
has been properly configured.
There's nothing incorrect with your program, it works just fine on my CDH5
Spark-on-YARN instance (and SHOW TABLES; inside Hive does show the
"persontable" created, after the program's completion).
On Tue, Dec 8, 2015 at 11:47 AM Divya Gehlot <divya.htco...@gmail.com>
wrote:
> Hi,
>
> I am new bee to Spark and using HDP 2.2 which comes with Spark 1.3.1
> I tried following code example
>
>> import org.apache.spark.sql.SQLContext
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>> import sqlContext.implicits._
>>
>> val personFile = "/user/hdfs/TestSpark/Person.csv"
>> val df = sqlContext.load(
>> "com.databricks.spark.csv",
>> Map("path" -> personFile, "header" -> "true", "inferSchema" ->
>> "true"))
>> df.printSchema()
>> val selectedData = df.select("Name", "Age")
>> selectedData.save("NewPerson.csv", "com.databricks.spark.csv")
>> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>> hiveContext.sql("CREATE TABLE IF NOT EXISTS PersonTable (Name STRING, Age
>> STRING)")
>> hiveContext.sql("LOAD DATA INPATH '/user/hdfs/NewPerson.csv' INTO TABLE
>> PersonTable")
>> hiveContext.sql("from PersonTable SELECT Name, Age
>> ").collect.foreach(println)
>
>
> I am able to access above table in HDFS
>
>> [hdfs@sandbox ~]$ hadoop fs -ls /user/hive/warehouse/persontable
>> Found 3 items
>> -rw-r--r-- 1 hdfs hdfs 0 2015-12-08 04:40
>> /user/hive/warehouse/persontable/_SUCCESS
>> -rw-r--r-- 1 hdfs hdfs 47 2015-12-08 04:40
>> /user/hive/warehouse/persontable/part-00000
>> -rw-r--r-- 1 hdfs hdfs 33 2015-12-08 04:40
>> /user/hive/warehouse/persontable/part-00001
>
>
> But when I try show tables in hive console ,I couldnt find the table.
>
>> hive> use default ;
>> OK
>> Time taken: 0.864 seconds
>> hive> show tables;
>> OK
>> dataframe_test
>> sample_07
>> sample_08
>> Time taken: 0.521 seconds, Fetched: 3 row(s)
>> hive> use xademo ;
>> OK
>> Time taken: 0.791 seconds
>> hive> show tables;
>> OK
>> call_detail_records
>> customer_details
>> recharge_details
>> Time taken: 0.256 seconds, Fetched: 3 row(s)
>
>
> Can somebody guide me to right direction ,if something is wrong with the
> code or I am unable to understand the concepts.
> Would really appreciate your help.
>
> Thanks,
> Divya
>
>
