Perhaps you also need to upgrade Scala?
Clay Stevens
From: Hanyu Huang <hhu...@liftoff.io>
Sent: Wednesday, 15 November, 2023 1:15 AM
To: user@spark.apache.org
Subject: The job failed when we upgraded from spark 3.3.1 to spark3.4.1
Caution, this email may be from a sender outside Wolters Kluwer. Verify the
sender and know the content is safe.
The version our job originally ran was spark 3.3.1 and Apache Iceberg to 1.2.0，
But since we upgraded to spark3.4.1 and Apache Iceberg to 1.3.1, jobs started
to fail frequently， We tried to upgrade only iceberg without upgrading spark,
and the job did not report an error.
Detailed description：
When we execute this function writing data to the iceberg table：
def appendToIcebergTable(targetTable: String, df: DataFrame): Unit = {
_logger.warn(s"Append data to $targetTable")
val (targetCols, sourceCols) = matchDFSchemaWithTargetTable(targetTable, df)
df.createOrReplaceTempView("_temp")
spark.sql(s"""
INSERT INTO $targetTable ($targetCols) SELECT $sourceCols FROM _temp
""")
_logger.warn(s"Done append data to $targetTable")
getIcebergLastAppendCountVerbose(targetTable)
}
The error is reported as follows:
Caused by: java.lang.AssertionError: assertion failed at
scala.Predef$.assert(Predef.scala:208) at
org.apache.spark.sql.execution.ColumnarToRowExec.<init>(Columnar.scala:72) ...
191 more
Read the source code and find that the error is reported here：
case class ColumnarToRowExec(child: SparkPlan) extends ColumnarToRowTransition
with CodegenSupport {
// supportsColumnar requires to be only called on driver side, see also
SPARK-37779.
assert(Utils.isInRunningSparkTask || child.supportsColumnar)
override def output: Seq[Attribute] = child.output
override def outputPartitioning: Partitioning = child.outputPartitioning
override def outputOrdering: Seq[SortOrder] = child.outputOrdering
But we can't find the root cause，So seek help from the community ，If more log
information is required, please let me know.
thanks
