chandu-1101 commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1683435824
I am trying to understand the schema issue. We first generate parquet files from Mongo. we prepare the dataframe on these parquet files, get schema, and give it to the dataframe created over CDC -`.json` files. So essentially it's the same schema with CDC, parquet. I reran the test, got the schema from the Iceberg and Parquet files. I couldn't find a reference to `long` -- exception was `Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long.` [iceberg-schema.txt](https://github.com/apache/iceberg/files/12376279/iceberg-schema.txt) [snapshot-schema.txt](https://github.com/apache/iceberg/files/12376286/snapshot-schema.txt) --Generated from the dataframe over the set of input partitioned parquet files. I suspect a bad row in the JSON. and tried to remove it as follows. But this also didn't work. ``` val cdcDf = sess.read .schema(snapshotDf.schema) .option("mode","PERMISSIVE") .option("dateformat","yyyy-MM-dd") .option("badRecordsPath","s3://bucket/inputs2/bad-records/").json("s3://p-crm-messaging-v2/inputs2/x11-partitioned/") // .option("mode", "DROPMALFORMED") cdcDf.createOrReplaceTempView("cdc") val _cdcDf =sess.sql(""" select * from cdc c1 where cdc_pk in ( select max(cdc_pk) from cdc c2 where _id.oid is not null and _id.oid !='' and c2.__created_date_=c1.__created_date_ group by _id.oid) """) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
