HyukjinKwon commented on code in PR #58615:
URL: https://github.com/apache/spark/pull/58615#discussion_r3994456188
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala:
##########
@@ -200,6 +200,24 @@ private[spark] object HiveUtils extends Logging {
.booleanConf
.createWithDefault(false)
+ val INITIALIZE_METASTORE_FORMAT_CLASSES =
+ buildConf("spark.sql.hive.initializeMetastoreFormatClasses")
+ .doc("When true, an InputFormat/OutputFormat class name stored in the
Hive metastore is " +
+ "resolved with its static initializer run at resolution time. When
false, the class is " +
+ "resolved without running its static initializer, which then runs
later when the format " +
+ "is instantiated for a scan/write. Only the class name is needed when
converting " +
+ "metastore metadata, so setting this to false avoids running a format
class's static " +
+ "initializer during a metadata operation. Note that the conversion
also runs when " +
+ "planning a scan or write and when inferring the schema of a Hive
serde table, so with " +
+ "false a format class whose static initializer fails no longer fails
fast on the driver " +
+ "at resolution time; the failure instead surfaces later on an executor
when the format " +
+ "is instantiated (as NoClassDefFoundError: Could not initialize class
...). Keep this " +
+ "true (the default) to preserve the fail-fast behavior.")
Review Comment:
Adopted your suggested doc in 384ea6b: scan instantiates on the driver at
split computation, write on an executor, the built-in Parquet/ORC reader never
instantiates the format, and the class is still loaded so a missing class still
fails at resolution time.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala:
##########
@@ -20,9 +20,41 @@ package org.apache.spark.sql.hive.client
import org.apache.hadoop.hive.metastore.api.FieldSchema
import org.apache.spark.{SparkFunSuite, SparkUnsupportedOperationException}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat,
CatalogTable, CatalogTableType}
+import org.apache.spark.sql.hive.{HiveUtils, StaticInitFlags,
StaticInitInputFormat}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
class HiveClientImplSuite extends SparkFunSuite {
+ test("SPARK-59330: toHiveTable skips the format class static initializer
when " +
+ "spark.sql.hive.initializeMetastoreFormatClasses is false") {
+ val table = CatalogTable(
+ identifier = TableIdentifier("t", Some("default")),
+ tableType = CatalogTableType.MANAGED,
+ storage = CatalogStorageFormat.empty.copy(
+ inputFormat = Some(classOf[StaticInitInputFormat].getName)),
+ schema = new StructType().add("a", "int"))
+
+ def toHiveTableWith(initialize: Boolean): Unit = {
+ val conf = new SQLConf()
+ conf.setConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES, initialize)
+ SQLConf.withExistingConf(conf) {
+ HiveClientImpl.toHiveTable(table)
+ }
+ }
+
+ // The false half must run first: class initialization is one-way per JVM.
Resolving the
+ // format class name without initializing it must not run the static
initializer.
+ toHiveTableWith(initialize = false)
+ assert(!StaticInitFlags.inputFormatInitialized)
+
+ // With initialization enabled (the default), resolving the class name
runs the initializer.
+ toHiveTableWith(initialize = true)
+ assert(StaticInitFlags.inputFormatInitialized)
Review Comment:
Done in 384ea6b. Added `StaticInitOutputFormat` (`HiveOutputFormat<Void,
Void>` whose methods throw) and `outputFormatInitialized` on `StaticInitFlags`,
put both formats on the test table storage, and assert both flags in each half
-- so `toOutputFormat` is now covered alongside `toInputFormat`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
