Hi,
I am able to make it in spark 2.3.0, If you can change the version to spark
2.3 it will be good , otherwise let me know , i 'll check on the Spark
version 2.1.0.
Following is the code for spark 2.3.0.
scala> var seq = Seq((10L,"Hello"),(10L,"Hi"))
seq: Seq[(Long, String)] = List((10,Hello), (10,Hi))
scala> seq.toDF("a","b")
res0: org.apache.spark.sql.DataFrame = [a: bigint, b: string]
scala> res0.show()
+---+-----+
| a| b|
+---+-----+
| 10|Hello|
| 10| Hi|
+---+-----+
scala> res0.printSchema
root
|-- a: long (nullable = false)
|-- b: string (nullable = true)
scala>
res0.write.option("path","/tmp/longpartition1").mode("overwrite").partitionBy("a").saveAsTable("default.longPartition")
hive> select * from longpartition
> ;
OK
Hello 10
Hi 10
Time taken: 0.356 seconds, Fetched: 2 row(s)
hive> describe longpartition;
OK
b string
a bigint
# Partition Information
# col_name data_type comment
a bigint
On Mon, Dec 16, 2019 at 11:05 AM SB M <[email protected]> wrote:
> spark version 2.1.0
>
> Regards,
> Sbm
>
> On Mon, 16 Dec, 2019, 10:04 HARSH TAKKAR, <[email protected]> wrote:
>
>> Please share the spark version you are using .
>>
>> On Fri, 13 Dec, 2019, 4:02 PM SB M, <[email protected]> wrote:
>>
>>> Hi All,
>>> Am trying to create a dynamic partition with external table on hive
>>> metastore using spark sql.
>>>
>>> when am trying to create a partition column data type as bigint,
>>> partition is not working even i tried with repair table. data is not shown
>>> when i ran sample query select * from table.
>>>
>>>
>>> but if i tried to create a dynamic partition with string as data type
>>> for partition its working fine. partition are working as expected.
>>>
>>>
>>> is there something am doing it wrong ??
>>>
>>>
>>> Regards,
>>> Sree
>>>
>>>
