Hi All, I had created an issue on Stackoverflow(linked below) a few months back about issues while handling bignumeric type values of BigQuery in Spark.
link <https://stackoverflow.com/questions/74719503/getting-error-while-reading-bignumeric-data-type-from-a-bigquery-table-using-apa> On Fri, Feb 24, 2023 at 3:54 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Nidhi, > > can you create a BigQuery table with a bignumeric and numeric column > types, add a few lines and try to read into spark. through DF > > and do > > > df.printSchema() > > df.show(5,False) > > > HTH > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Fri, 24 Feb 2023 at 02:47, nidhi kher <kherni...@gmail.com> wrote: > >> Hello, >> >> >> I am facing below issue in pyspark code: >> >> We are running spark code using dataproc serverless batch in google cloud >> platform. Spark code is causing issue while writing the data to bigquery >> table. In bigquery table , few of the columns have datatype as bignumeric >> and spark code is changing the datatype from bignumeric to numeric while >> writing the data. We need datatype to be kept as bignumeric only as we need >> data of 38,20 precision. >> >> >> Can we cast a column to bignumeric in spark sql dataframe like below code >> for decimal: >> >> >> df= spark.sql("""SELECT cast(col1 as decimal(38,20)) as col1 from >> table1""") >> >> Spark version :3.3 >> >> Pyspark version : 1.1 >> >> >> Regards, >> >> Nidhi >> >
