Re: How to insert a dataframe as a static partition to a partitioned table
Thanks Vadim. But I am looking for an API either in DataSet, DataFrame, or DataFrameWriter etc. The way you suggested can be done via a query like spark.sql(""" ALTER TABLE `table` ADD PARTITION (partcol=1) LOCATION '/path/to/your/dataset' """), and before that I write it to a specified location first. On Thu, Jul 20, 2017 at 10:56 AM, Vadim Semenovwrote: > This should work: > ``` > ALTER TABLE `table` ADD PARTITION (partcol=1) LOCATION > '/path/to/your/dataset' > ``` > > On Wed, Jul 19, 2017 at 6:13 PM, ctang wrote: > >> I wonder if there are any easy ways (or APIs) to insert a dataframe (or >> DataSet), which does not contain the partition columns, as a static >> partition to the table. For example, >> The DataSet with columns (col1, col2) will be inserted into a table (col1, >> col2) partitioned by column partcol as a static partition with partspec >> (partcol =1). >> >> Thanks >> >> >> >> -- >> View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/How-to-insert-a-dataframe-as-a-static- >> partition-to-a-partitioned-table-tp28882.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >
Re: How to insert a dataframe as a static partition to a partitioned table
This should work: ``` ALTER TABLE `table` ADD PARTITION (partcol=1) LOCATION '/path/to/your/dataset' ``` On Wed, Jul 19, 2017 at 6:13 PM, ctangwrote: > I wonder if there are any easy ways (or APIs) to insert a dataframe (or > DataSet), which does not contain the partition columns, as a static > partition to the table. For example, > The DataSet with columns (col1, col2) will be inserted into a table (col1, > col2) partitioned by column partcol as a static partition with partspec > (partcol =1). > > Thanks > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/How-to-insert-a-dataframe-as-a- > static-partition-to-a-partitioned-table-tp28882.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: How to insert a dataframe as a static partition to a partitioned table
Not sure about the writer api, but you could always register a temp table for that dataframe and execute insert hql. On Thu, Jul 20, 2017 at 6:13 AM, ctangwrote: > I wonder if there are any easy ways (or APIs) to insert a dataframe (or > DataSet), which does not contain the partition columns, as a static > partition to the table. For example, > The DataSet with columns (col1, col2) will be inserted into a table (col1, > col2) partitioned by column partcol as a static partition with partspec > (partcol =1). > > Thanks > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/How-to-insert-a-dataframe-as-a- > static-partition-to-a-partitioned-table-tp28882.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >