nzw921rx opened a new pull request, #11791: URL: https://github.com/apache/seatunnel/pull/11791
## Purpose Full connector E2E runs are increasingly dominated by a few long shards. In recent runs, parts 3 and 7 took about 170 minutes while parts 1 and 2 finished in about 60-65 minutes. Iceberg and HBase were also grouped into one approximately 140-minute job, so the overall workflow remained constrained by long-tail jobs even though Java 8 and Java 11 already run concurrently. ## Changes - Split Iceberg and HBase into independent jobs so their Java 8/11 matrices can run concurrently instead of serializing both suites in one shard. - Make both dedicated jobs run for API/engine changes and direct changes to their own E2E modules. - Exclude Iceberg and HBase centrally from both full and updated-module shared shards, avoiding duplicate execution regardless of POM module order. - Replace order-dependent modulo assignment for the seven full connector shards with a stable seeded hash. - Select the seed from the module durations in Actions run 31611542978; the resulting historical estimate is 79.1-89.6 minutes per shard, compared with the previous approximately 64-172 minute spread. - Keep the runtime data out of the maintained configuration: the implementation stores only one stable seed. Adding or removing a connector does not move existing modules and requires no duration-table update. - Align the seven balanced shard timeouts at 150 minutes, with 120 minutes for Iceberg and 150 minutes for HBase. - Add regression tests for unique assignment, order independence, stable assignment when a module is added, and dedicated-suite exclusions. ## Historical evidence Recent successful Java 8/11 connector jobs showed approximately: - part 1: 64 minutes - part 2: 61-65 minutes - part 3: 165-170 minutes - part 4: 98-101 minutes - part 5: 119-125 minutes - part 6: 101-103 minutes - part 7: 171-172 minutes - combined Iceberg + HBase: 138-141 minutes The Reactor Summary also showed Iceberg around 44 minutes and HBase around 90 minutes, which makes independent jobs preferable to a combined serial job. ## Validation - `./mvnw spotless:apply` - `python3 -m unittest discover -s tools/update_modules_check -p "test_*.py"` (19 tests passed) - Parsed `.github/workflows/backend.yml` as YAML and verified both dedicated jobs are present and the old combined job is absent - Executed the exact module-discovery and `sub_it_module` command chain used by the workflow - Verified all 53 shared connector E2E modules are assigned exactly once across parts 1-7 - Verified Iceberg/HBase-only updates are removed from updated shared shards, while mixed updates retain the remaining connector modules - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
