voonhous opened a new pull request, #19932: URL: https://github.com/apache/hudi/pull/19932
### Describe the issue this Pull Request addresses Part of #19524, stacked on #19928 (phase 4 of the Scala parallel work). With dml-1, dml-2 and other-1 running suites two at a time, the run's longest job became scala-other-2 at 35 to 37 minutes: the only Scala shard still serial, because its packages mix 33 suites on `HoodieSparkSqlTestBase` (26.5 of its 28.4 minutes of tests) with 10 suites that build their own SparkContext (1.9 minutes). ### Summary and Changelog - New ScalaTest tag annotation `SharedSessionSuite` (`@TagAnnotation`, `@Inherited`) on `HoodieSparkSqlTestBase`, so every suite on that base carries the tag without being listed, including the procedure base's subclasses. - scala-other-2 runs only the tagged suites of its packages (`-DtagsToInclude`), in shared-session mode two at a time, the same flags as the other three SQL shards. - The untagged suites of the same packages (`-DtagsToExclude`) run as an extra step of test-spark-java17-ftb, the shortest Java 17 job. A new suite lands on this serial side unless it extends the base. - Three things the first runs exposed, each its own commit: the four index lookup suites unpersist every RDD in the context and assert it holds none, so their base runs alone (`ExclusiveSuite`); hudi-spark's pom pinned the scalatest `tagsToExclude`, which silently beat the `-D` flag, so it is now the property `hudi.scalatest.tagsToExclude` with the same default; and two stream-close gaps in hudi-hadoop-common that Spark's DebugFilesystem exposed once it saw Hudi's reads: `HadoopFSUtils.getFSDataInputStream` never closed the FSDataInputStream object a filesystem returned, only the inner stream it rebuffered (an accounting miss), and `InLineFsDataInputStream` had no `close` at all, so every inline HFile read of a metadata table log block leaked the outer file handle until GC (a real leak). Each has a unit test, and each is also up on its own for review, #19935 and #19936; this PR carries copies until they merge and drops them on its next rebase. Job count unchanged at fourteen. Measurement, three samples: scala-other-2's scalatest wall clock against its serial 28:24, the ftb job against its 21 to 28, and the run's longest job against 35 to 37. | job | before | sample 1 | sample 2 | sample 3 | |---|---|---|---|---| | scala-other-2 tests | 28:24 serial | [23:34](https://github.com/apache/hudi/actions/runs/34748622752/job/103700994798), job 31 | [23:44](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2/job/103703547405), job 31 | [22:56](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3/job/103707769911), job 30 | | ftb job total | 21 to 28 | [30](https://github.com/apache/hudi/actions/runs/34748622752/job/103700994761), the new step 2:27 | [30](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2/job/103703547479), the step 2:26 | [35](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3/job/103707769878), the step 2:49 on a slow runner (its Java step took 25 against 21) | | run's longest job | 35 to 37, scala-other-2 | [35, fta](https://github.com/apache/hudi/actions/runs/34748622752) | [33, fta](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2) | [39, ut-fat-and-ftc](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3) | Sample 1 notes: 151 tests on other-2 plus 88 on ftb's step are exactly the 239 names of the serial job. other-2 at two threads keeps only 1.2 threads busy because TestSecondaryIndex, TestExpressionIndex and the lookup suites run alone under the exclusive lock; that is its floor now. Runner-minutes for the runs: 514, 491, 512. Verdict after three samples: other-2 goes from 28:24 of tests and a 35 to 37 minute job to 22:56 to 23:44 and 30 to 31, every sample below serial, the same 239 test names split 151 and 88 each time. ftb absorbs the 88 own-context tests at 2:26 to 2:49 plus about a minute of Maven, and lands at 30 to 35 depending on the runner. The run's longest job is now a Java job every time: fta, fta, ut-fat-and-ftc at 33 to 39. The Scala shards no longer set the wall clock, which is where this stack stops. <details> <summary>Why the stream leaks only showed up now</summary> Spark's `StreamTest` suites install `DebugFilesystem` for `file:` and assert at teardown that every stream it handed out was closed. Hadoop caches one FileSystem per scheme for the whole JVM, keyed by scheme, not by implementation. In the serial other-2 job the SQL suites ran first and cached a plain LocalFileSystem, so DebugFilesystem never saw Hudi's reads and the check had nothing to check. In the new ftb step the two `StreamTest` suites are the first to touch `file:`, the check finally covers Hudi's log and HFile reads, and TestStreamingSource aborted with "26 possibly leaked file streams". 23 of those were the rebuffering accounting miss; the remaining 3 were the inline filesystem never closing the outer file, which is a real handle leak on every inline HFile read. </details> <details> <summary>Why a tag rather than package filters or a suite list</summary> The two families live in the same packages, and the plugin's `wildcardSuites` cannot exclude. An explicit list of the ten own-context suites would silently route a new own-context suite into the shared-session job, where it would break. The inherited tag makes the shared-session side opt-in by base class, verified locally against the project's ScalaTest: a Java annotation with `@TagAnnotation` and `@Inherited` on a base class is seen by `-n` and `-l` through two levels of subclassing. </details> ### Impact CI configuration and test infrastructure, plus two stream-close fixes in hudi-hadoop-common, one of them a real file-handle leak on inline HFile reads. ### Risk Level low. Off by default like the rest of the stack; the ten own-context suites keep a private SparkContext. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
