Hi Varun, I am no expert on Snowflake, however, the issue you are facing, particularly if it involves data trimming in a COPY statement and potential data mismatch, is likely related to how Snowflake handles data ingestion rather than being directly tied to PySpark. The COPY command in Snowflake is used to load data from external files (like those in s3) into Snowflake tables. Possible causes for data truncation or mismatch could include differences in data types, column lengths, or encoding between your source data and the Snowflake table schema. It could also be related to the way your PySpark application is formatting or providing data to Snowflake.
Check these - Schema Matching: Ensure that the data types, lengths, and encoding of the columns in your Snowflake table match the corresponding columns in your PySpark DataFrame. - Column Mapping: Explicitly map the columns in your PySpark DataFrame to the corresponding columns in the Snowflake table during the write operation. This can help avoid any implicit mappings that might be causing issues. 1. HTH Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 9 Feb 2024 at 13:06, Varun Shah <varunshah100...@gmail.com> wrote: > Hi Team, > > We currently have implemented pyspark spark-streaming application on > databricks, where we read data from s3 and write to the snowflake table > using snowflake connector jars (net.snowflake:snowflake-jdbc v3.14.5 and > net.snowflake:spark-snowflake v2.12:2.14.0-spark_3.3) . > > Currently facing an issue where if we give a large number of columns, it > trims the data in a copy statement, thereby unable to write to the > snowflake as the data mismatch happens. > > Using databricks 11.3 LTS with Spark 3.3.0 and Scala 2.12 version. > > Can you please help on how I can resolve this issue ? I tried searching > online, but did not get any such articles. > > Looking forward to hearing from you. > > Regards, > Varun Shah > > >
