Hi All I am writing a bigquery connector here <http://github.com/samelamin/spark-bigquery> and I am getting a strange error with schemas being overwritten when a dataframe is passed over to the Sink
for example the source returns this StructType WARN streaming.BigQuerySource: StructType(StructField(customerid,LongType,true), and the sink is recieving this StructType WARN streaming.BigQuerySink: StructType(StructField(customerid,StringType,true) Any idea why this might be happening? I dont have infering schema on spark.conf.set("spark.sql.streaming.schemaInference", "false") I know its off by default but I set it just to be sure So completely lost to what could be causing this Regards Sam