Hi, I have testest few JDBC BigQuery providers like Progress Direct and Simba but none of them seem to work properly through Spark.
The only way I can read and write to BigQuery is through Spark BigQuery API using the following scenario spark-shell --jars=gs://spark-lib/bigquery/spark-bigquery-latest.jar Using the following JDBC connection to read val BQDF = spark.read. format("bigquery"). option("credentialsFile",jsonKeyFile). option("project", projectId). option("parentProject", projectId). option("dataset", targetDataset). option("table", targetTable). option("partitionColumn", partitionColumn). option("lowerBound", lowerBound). option("upperBound", upperBound). option("numPartitions", numPartitions). load() and for write rsBatch. write. format("bigquery"). mode(org.apache.spark.sql.SaveMode.Append). option("table", fullyQualifiedOutputTableId). save() Appreciate any comments if someone has managed to make this work through any third party JDBC drivers. Regards, Mich *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.