I was expecting to get the same results with both:
dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
and
dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
"my_table")
In the first example, it behaves as expected. It creates a new table and
populates it with the rows from DataFrame.
In the second case, I get exception:
org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not
allow create table as select.
Looking at the Spark source, it looks like there is a completely separate
implementation for format("jdbc") and for jdbc(...).
I find that confusing. Unfortunately documentation is rather sparse and one
finds this discrepancy only through trial and error.
Is there a plan to deprecate one of the forms ? Or to allow same
functionality for both ?
I tried both 1.6 and 2.0-preview
--
Dragiša Krsmanović | Platform Engineer | Ticketfly
dragi...@ticketfly.com
@ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
facebook.com/ticketfly
