[ https://issues.apache.org/jira/browse/SPARK-39605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manoj Chandrashekar updated SPARK-39605: ---------------------------------------- Attachment: image-2022-06-27-11-00-50-119.png > PySpark df.count() operation works fine on DBR 7.3 LTS but fails in DBR 10.4 > LTS > -------------------------------------------------------------------------------- > > Key: SPARK-39605 > URL: https://issues.apache.org/jira/browse/SPARK-39605 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 3.2.1 > Reporter: Manoj Chandrashekar > Priority: Major > Attachments: image-2022-06-27-11-00-50-119.png > > > I have a job that infers schema from mongodb and does operations such as > flattening and unwinding because there are nested fields. After performing > various transformations, finally when the count() is performed, it works > perfectly fine in databricks runtime version 7.3 LTS but fails to perform the > same in 10.4 LTS. > *Below is the image that shows successful run in 7.3 LTS:* > !https://docs.microsoft.com/answers/storage/attachments/215035-image.png|width=672,height=80! > *Below is the image that shows failure in 10.4 LTS:* > !https://docs.microsoft.com/answers/storage/attachments/215026-image.png|width=668,height=69! > And I have validated that there is no field in our schema that has NullType. > In fact when the schema was inferred, there were Null & void type fields > which were converted to string using my UDF. This issue will persists even > when I infer schema on complete dataset, that is, samplePoolSize is on full > data set. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org