[
https://issues.apache.org/jira/browse/SPARK-31001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598582#comment-17598582
]
Kevin Appel commented on SPARK-31001:
-------------------------------------
I am trying to get SparkR and Sparklyr to work with this and did some more
testing and found out for the pyspark you don't actually need to pass it that
partition columns, at least not in spark 3.2.1, doing the recover partitions is
grabbing it all, there is a also a builtin for doing the recover
spark.catalog.createTable("kevin.ktest2", "/user/kevin/ktest2")
spark.catalog.recoverPartitions("kevin.ktest2")
The SparkR has a similar items
createTable("kevin.ktest3", path="/user/kevin/ktest3")
recoverPartitions("kevin.ktest3")
I don't see there is an easy way to do this in sparklyr
Maybe this is the official way?
> Add ability to create a partitioned table via catalog.createTable()
> -------------------------------------------------------------------
>
> Key: SPARK-31001
> URL: https://issues.apache.org/jira/browse/SPARK-31001
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Nicholas Chammas
> Priority: Minor
>
> There doesn't appear to be a way to create a partitioned table using the
> Catalog interface.
> In SQL, however, you can do this via {{{}CREATE TABLE ... PARTITIONED BY{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
