Neilk1021 opened a new pull request, #6238: URL: https://github.com/apache/texera/pull/6238
### What changes were proposed in this PR? Closes #6063 by completely refactoring the `MockTexeraDB` infrastructure to transition from a distributed process architecture to a centralized, shared-singleton architecture. 1. **High-Performance Shared Singleton Architecture:** Refactored the core setup by splitting the original `MockTexeraDB` trait into a companion `object MockTexeraDB` singleton and a lightweight `trait MockTexeraDB` interface wrapper. The background `EmbeddedPostgres` database engine process is now booted **exactly once** per JVM lifecycle via synchronized initialization. Individual test suites dynamically provision an isolated schema sandbox via cheap `CREATE DATABASE` queries instead of spawning heavy process-level instances. 2. **DDL Caching and Regex Optimization:** Removed massive runtime file I/O and processing bottlenecks. The full-text search index string regex modifications and SQL script cleaning are handled once and cached globally in memory (`ddlScript`), preventing parallel test suites from thrashing disk resources over the same `sql/texera_ddl.sql` file. 3. **Robust Connection Lifecycle Management:** Addressed multi-threaded race conditions crashing asynchronous multipart uploads (`PSQLException: Cannot commit when autoCommit is enabled`). Refactored `SqlServer.withTransaction` to safely intercept test configurations, bypassing rigid jOOQ transaction managers to manually cycle and restore `autoCommit` states cleanly on the active connection handle without leaking parameters across tests. ### Performance Impact Moving database instantiation, parsing, and regex evaluation out of the distributed trait lifecycle and into a centralized static memory layer resulted in a dramatic reduction in test suite runtime. * **Main Branch Baseline / Prior Implementation:** 5 minutes 40 seconds * **This PR:** **4 minutes 5 seconds** (Total execution time reduced by ~28% locally). ### Any related issues, documentation, discussions? Follow up on #4525 where a refactor of `MockTexeraDB` was needed to allow for future concurrent execution of test suites. Structural baseline was cherry-picked and heavily adapted from #4527, specifically commit `bd93161` written by @Yicong-Huang. ### How was this PR tested? Verified by running full, clean compilations and local stress tests on JDK 17 to validate that the structural split preserves total API backward compatibility across the codebase. Additionally validated that high-concurrency race conditions (such as the asynchronous multi-threaded tests inside `DatasetResourceSpec`) converge gracefully without experiencing deadlocks, thread crashes, or connection state leakage. All 1,600+ test suite assertions across the system are green. Run configuration command used to validate the project modules: ```bash sbt clean compile test ``` ### Was this PR authored or co-authored using generative AI tooling? Yes. Generative AI (Gemini) was utilized as an external assistant to analyze complex multi-threaded JDBC/jOOQ connection trace errors and help structure the safe connection lifecycle fallback logic in SqlServer.scala. No direct IDE-integrated automation tools were used to auto-generate codebase files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
