voonhous opened a new pull request, #19932:
URL: https://github.com/apache/hudi/pull/19932

   ### Describe the issue this Pull Request addresses
   
   Part of #19524, stacked on #19928 (phase 4 of the Scala parallel work). With 
dml-1, dml-2 and other-1 running suites two at a time, the run's longest job 
became scala-other-2 at 35 to 37 minutes: the only Scala shard still serial, 
because its packages mix 33 suites on `HoodieSparkSqlTestBase` (26.5 of its 
28.4 minutes of tests) with 10 suites that build their own SparkContext (1.9 
minutes).
   
   ### Summary and Changelog
   
   - New ScalaTest tag annotation `SharedSessionSuite` (`@TagAnnotation`, 
`@Inherited`) on `HoodieSparkSqlTestBase`, so every suite on that base carries 
the tag without being listed, including the procedure base's subclasses.
   - scala-other-2 runs only the tagged suites of its packages 
(`-DtagsToInclude`), in shared-session mode two at a time, the same flags as 
the other three SQL shards.
   - The untagged suites of the same packages (`-DtagsToExclude`) run as an 
extra step of test-spark-java17-ftb, the shortest Java 17 job. A new suite 
lands on this serial side unless it extends the base.
   
   - Three things the first runs exposed, each its own commit: the four index 
lookup suites unpersist every RDD in the context and assert it holds none, so 
their base runs alone (`ExclusiveSuite`); hudi-spark's pom pinned the scalatest 
`tagsToExclude`, which silently beat the `-D` flag, so it is now the property 
`hudi.scalatest.tagsToExclude` with the same default; and two stream-close gaps 
in hudi-hadoop-common that Spark's DebugFilesystem exposed once it saw Hudi's 
reads: `HadoopFSUtils.getFSDataInputStream` never closed the FSDataInputStream 
object a filesystem returned, only the inner stream it rebuffered (an 
accounting miss), and `InLineFsDataInputStream` had no `close` at all, so every 
inline HFile read of a metadata table log block leaked the outer file handle 
until GC (a real leak). Each has a unit test, and each is also up on its own 
for review, #19935 and #19936; this PR carries copies until they merge and 
drops them on its next rebase.
   
   Job count unchanged at fourteen.
   
   Measurement, three samples: scala-other-2's scalatest wall clock against its 
serial 28:24, the ftb job against its 21 to 28, and the run's longest job 
against 35 to 37.
   
   | job | before | sample 1 | sample 2 | sample 3 |
   |---|---|---|---|---|
   | scala-other-2 tests | 28:24 serial | 
[23:34](https://github.com/apache/hudi/actions/runs/34748622752/job/103700994798),
 job 31 | 
[23:44](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2/job/103703547405),
 job 31 | 
[22:56](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3/job/103707769911),
 job 30 |
   | ftb job total | 21 to 28 | 
[30](https://github.com/apache/hudi/actions/runs/34748622752/job/103700994761), 
the new step 2:27 | 
[30](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2/job/103703547479),
 the step 2:26 | 
[35](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3/job/103707769878),
 the step 2:49 on a slow runner (its Java step took 25 against 21) |
   | run's longest job | 35 to 37, scala-other-2 | [35, 
fta](https://github.com/apache/hudi/actions/runs/34748622752) | [33, 
fta](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2) | [39, 
ut-fat-and-ftc](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3)
 |
   
   Sample 1 notes: 151 tests on other-2 plus 88 on ftb's step are exactly the 
239 names of the serial job. other-2 at two threads keeps only 1.2 threads busy 
because TestSecondaryIndex, TestExpressionIndex and the lookup suites run alone 
under the exclusive lock; that is its floor now. Runner-minutes for the runs: 
514, 491, 512.
   
   Verdict after three samples: other-2 goes from 28:24 of tests and a 35 to 37 
minute job to 22:56 to 23:44 and 30 to 31, every sample below serial, the same 
239 test names split 151 and 88 each time. ftb absorbs the 88 own-context tests 
at 2:26 to 2:49 plus about a minute of Maven, and lands at 30 to 35 depending 
on the runner. The run's longest job is now a Java job every time: fta, fta, 
ut-fat-and-ftc at 33 to 39. The Scala shards no longer set the wall clock, 
which is where this stack stops.
   
   <details>
   <summary>Why the stream leaks only showed up now</summary>
   
   Spark's `StreamTest` suites install `DebugFilesystem` for `file:` and assert 
at teardown that every stream it handed out was closed. Hadoop caches one 
FileSystem per scheme for the whole JVM, keyed by scheme, not by 
implementation. In the serial other-2 job the SQL suites ran first and cached a 
plain LocalFileSystem, so DebugFilesystem never saw Hudi's reads and the check 
had nothing to check. In the new ftb step the two `StreamTest` suites are the 
first to touch `file:`, the check finally covers Hudi's log and HFile reads, 
and TestStreamingSource aborted with "26 possibly leaked file streams". 23 of 
those were the rebuffering accounting miss; the remaining 3 were the inline 
filesystem never closing the outer file, which is a real handle leak on every 
inline HFile read.
   </details>
   
   <details>
   <summary>Why a tag rather than package filters or a suite list</summary>
   
   The two families live in the same packages, and the plugin's 
`wildcardSuites` cannot exclude. An explicit list of the ten own-context suites 
would silently route a new own-context suite into the shared-session job, where 
it would break. The inherited tag makes the shared-session side opt-in by base 
class, verified locally against the project's ScalaTest: a Java annotation with 
`@TagAnnotation` and `@Inherited` on a base class is seen by `-n` and `-l` 
through two levels of subclassing.
   </details>
   
   ### Impact
   
   CI configuration and test infrastructure, plus two stream-close fixes in 
hudi-hadoop-common, one of them a real file-handle leak on inline HFile reads.
   
   ### Risk Level
   
   low. Off by default like the rest of the stack; the ten own-context suites 
keep a private SparkContext.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to