Hi Mich I wrote a connector to make it easier to connect Bigquery and Spark
Have a look here https://github.com/samelamin/spark-bigquery/ Your feedback is always welcome Kind Regards Sam On Tue, Dec 18, 2018 at 7:46 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks Jorn. I will try that. Requires installing sbt etc on ephemeral > compute server in Google Cloud to built an uber jar file. > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Tue, 18 Dec 2018 at 11:16, Jörn Franke <jornfra...@gmail.com> wrote: > >> Maybe the guava version in your spark lib folder is not compatible (if >> your Spark version has a guava library)? In this case i propose to create a >> fat/uber jar potentially with a shaded guava dependency. >> >> Am 18.12.2018 um 11:26 schrieb Mich Talebzadeh <mich.talebza...@gmail.com >> >: >> >> Hi, >> >> I am writing a small test code in spark-shell with attached jar >> dependencies >> >> spark-shell --jars >> /home/hduser/jars/bigquery-connector-0.13.4-hadoop3.jar,/home/hduser/jars/gcs-connector-1.9.4-hadoop3.jar,/home/hduser/jars/other/guava-19.0.jar,/home/hduser/jars/google-api-client-1.4.1-beta.jar,/home/hduser/jars/google-api-client-json-1.2.3-alpha.jar,/home/hduser/jars/google-api-services-bigquery-v2-rev20181202-1.27.0.jar >> >> to read an already existing table in Google BigQuery as follows: >> >> import com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration >> import com.google.cloud.hadoop.io.bigquery.BigQueryFileFormat >> import com.google.cloud.hadoop.io.bigquery.GsonBigQueryInputFormat >> import >> com.google.cloud.hadoop.io.bigquery.output.BigQueryOutputConfiguration >> import >> com.google.cloud.hadoop.io.bigquery.output.IndirectBigQueryOutputFormat >> import com.google.gson.JsonObject >> import org.apache.hadoop.io.LongWritable >> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat >> // Assumes you have a spark context (sc) -- running from spark-shell REPL. >> // Marked as transient since configuration is not Serializable. This >> should >> // only be necessary in spark-shell REPL. >> @transient >> val conf = sc.hadoopConfiguration >> // Input parameters. >> val fullyQualifiedInputTableId = "axial-glow-224522.accounts.ll_18740868" >> val projectId = conf.get("fs.gs.project.id") >> val bucket = conf.get("fs.gs.system.bucket") >> // Input configuration. >> conf.set(BigQueryConfiguration.PROJECT_ID_KEY, projectId) >> conf.set(BigQueryConfiguration.GCS_BUCKET_KEY, bucket) >> BigQueryConfiguration.configureBigQueryInput(conf, >> fullyQualifiedInputTableId) >> >> The problem I have is that even after loading jars with spark-shell --jar >> >> I am getting the following error at the last line >> >> scala> BigQueryConfiguration.configureBigQueryInput(conf, >> fullyQualifiedInputTableId) >> >> java.lang.NoSuchMethodError: >> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V >> at >> com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68) >> at >> com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration.configureBigQueryInput(BigQueryConfiguration.java:260) >> ... 49 elided >> >> It says it cannot find method >> >> java.lang.NoSuchMethodError: >> com.google.common.base.Preconditions.checkArgument >> >> but I checked it and it is in the following jar file >> >> jar tvf guava-19.0.jar| grep common.base.Preconditions >> 5249 Wed Dec 09 15:58:14 UTC 2015 >> com/google/common/base/Preconditions.class >> >> I have used different version of guava jar files but none works! >> >> The code is based on the following: >> >> >> https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example >> >> Thanks >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >>
