wombatu-kun commented on code in PR #19926:
URL: https://github.com/apache/hudi/pull/19926#discussion_r4601702426
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/HoodieSparkSqlTestBase.scala:
##########
@@ -118,23 +157,68 @@ class HoodieSparkSqlTestBase extends FunSuite with
BeforeAndAfterAll {
// it, so a throwing or non-withRecordType INMEMORY test would
otherwise leak state here.
// Runs before the catalog cleanup so it holds even if a drop throws.
HoodieInMemoryHashIndex.clear()
- val catalog = spark.sessionState.catalog
- catalog.listDatabases().foreach { db =>
- catalog.listTables(db).foreach { table =>
- catalog.dropTable(table, true, true)
- }
- }
+ dropSuiteTables()
}
)
}
+ /**
+ * Shared mode: scalatest runs each suite on its own thread and Hudi reads
SparkSession.active in a few
+ * places (HoodieCatalog captures it when the session's catalog is first
built, BaseProcedure per CALL),
+ * so pin this suite's child session to the test thread and fail fast if the
session's HoodieCatalog was
+ * built against another session.
+ */
+ private def bindSuiteSession(): Unit = {
+ SparkSession.setActiveSession(spark)
+
spark.sessionState.catalogManager.catalog(CatalogManager.SESSION_CATALOG_NAME)
match {
+ case hoodieCatalog: HoodieCatalog =>
+ assert(hoodieCatalog.spark eq spark,
+ s"HoodieCatalog of ${getClass.getSimpleName} is bound to another
SparkSession")
+ case _ =>
+ }
+ }
+
+ /**
+ * Drops the tables a test left behind. Per-suite mode owns the whole
catalog. Shared mode shares the
+ * external catalog with every other suite in the JVM, so it drops only this
suite's generateTableName
+ * tables plus any table whose name is not of that form (fixed names and
temp views); under serial
+ * execution those can only come from the test that just ran. Fixed names
are renamed in a later step.
+ */
+ private def dropSuiteTables(): Unit = {
+ val catalog = spark.sessionState.catalog
+ catalog.listDatabases().foreach { db =>
Review Comment:
`dropSuiteTables` enumerates only external-catalog databases, leaving the
global temporary views created by `TestCopyToTempViewProcedure` alive for the
rest of the shared JVM. Please drop owned global views explicitly or clean them
up in that test's `finally`.
##########
.github/workflows/bot.yml:
##########
@@ -1222,14 +1224,14 @@ jobs:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_MODULES: ${{ matrix.sparkModules }}
run:
- mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_DML2_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
+ mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_DML2_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS
-Djacoco.skip=false -Dhudi.spark.test.sharedSession=true
Review Comment:
Enabling shared mode for `dml-2` keeps
`TestInsertTable4.StageParallelismListener` registered after its suite
finishes, so later suites still trigger its assertions and latch mutations.
Please retain and remove each listener in `finally` before enabling this shard.
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java:
##########
@@ -452,10 +452,27 @@ public HoodieFileFormat getFormat() {
*/
@Override
public List<GenericRecord> readAvroRecords(HoodieStorage storage,
StoragePath filePath) {
+ // A schema-less read takes the schema from the file; the copy drops any
read schema or
+ // projection a previous caller left on the shared configuration.
+ Configuration conf = storage.getConf().unwrapCopyAs(Configuration.class);
Review Comment:
The `readAvroRecords` fix has no regression test for preserving the caller's
`Configuration` or ignoring stale read-schema/projection settings on a
schema-less read. Please add a focused `TestParquetUtils` case that reuses one
configuration across schema-specific and schema-less reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]