Hi,
I am trying to write to a big query table via the bq-spark-connector using
the below 2 methods:
Ref Page: https://github.com/GoogleCloudDataproc/spark-bigquery-connector
<https://github.com/GoogleCloudDataproc/spark-bigquery-connector>
1. ==> df.write.format("bigquery").save("dataset.table")
2. ==> df.write.format("bigquery").option("writeMethod",
"direct").save("dataset.table")
execution time of both the above methods are kind of the same.
however when I try to create the table directly in the gcp console it runs
in a fraction of the time taken by the above 2 methods.
Are there some optimizations possible to create the dataset in BQ via spark
which may be comparable to the native BQ experience?
Thanks & Regards,
Abhinav Ranjan
