Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15072#discussion_r82737858 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -53,7 +53,15 @@ import org.apache.spark.util.Utils private[sql] object Dataset { def apply[T: Encoder](sparkSession: SparkSession, logicalPlan: LogicalPlan): Dataset[T] = { - new Dataset(sparkSession, logicalPlan, implicitly[Encoder[T]]) + val encoder = implicitly[Encoder[T]] + if (encoder.clsTag.runtimeClass == classOf[Row]) { + // We should use the encoder generated from the executed plan rather than the existing + // encoder for DataFrame because the types of columns can be varied due to widening types. + // See SPARK-17123. This is a bit hacky. Maybe we should find a better way to do this. + ofRows(sparkSession, logicalPlan).asInstanceOf[Dataset[T]] + } else { + new Dataset(sparkSession, logicalPlan, encoder) + } --- End diff -- Ah-ha, it seems we are widening types for set operators[1]. ```scala val dates = Seq( (new Date(0), BigDecimal.valueOf(1), new Timestamp(2)), (new Date(3), BigDecimal.valueOf(4), new Timestamp(5)) ).toDF("date", "timestamp", "decimal") val widenTypedRows = Seq( (new Timestamp(2), 10.5D, "string") ).toDF("date", "timestamp", "decimal") dates.printSchema() dates.except(widenTypedRows).toDF().printSchema() ``` prints ``` root |-- date: date (nullable = true) |-- timestamp: decimal(38,18) (nullable = true) |-- decimal: timestamp (nullable = true) root |-- date: timestamp (nullable = true) |-- timestamp: double (nullable = true) |-- decimal: string (nullable = true) ``` [1]https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L232-L249
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org