sreejasahithi commented on PR #11509: URL: https://github.com/apache/ozone/pull/11509#issuecomment-6342672798
<html><head></head><body><h3><span>Problem with the First Version of This PR</span></h3><p><span>The initial compose wiring used </span><code><span>execute_s3_smoketests()</span></code><span> in </span><code><span>testlib.sh</span></code><span>, which ran the full </span><code><span>smoketest/s3/</span></code><span> tree </span><strong><span>twice per call</span></strong><span> (</span><code><span>-v AWS_CLI:aws</span></code><span>, then </span><code><span>-v AWS_CLI:aws2</span></code><span>). Every compose leg that already ran </span><code><span>s3/</span></code><span> once now ran it twice on the </span><strong><span>same cluster configuration</span></strong><span>.</span></p><p><span>This causes approximately </span><strong><span>30 minutes of extra wall-clock time</span></strong></p> ### Why Not the Parallel Matrix ([#11569](https://github.com/apache/ozone/pull/11569))? [#11569](https://github.com/apache/ozone/pull/11569) tried to recover wall-clock time by splitting acceptance into matrix legs (`OZONE_AWS_CLI=aws|aws2`). That restores **per-job duration**, but still runs the full S3 suite **twice** in CI and adds **concurrent runner load** on the shared Apache queue. Review feedback was to avoid that plumbing and a full matrix, we should **not pay 2× the total work**, whether sequentially or in parallel. ### Proposed Approach: Partition CLI Across Existing Compose Legs (OzoneFS-Style) We already split coverage across **different acceptance environments** using Robot variables instead of rerunning the same suite everywhere. For example, OzoneFS uses `SCHEME` / `BUCKET_TYPE` across different legs: - `ozone/test.sh` — `SCHEME:o3fs`, `BUCKET_TYPE:bucket` - `ozone-ha/test.sh` — `SCHEME:ofs`, `BUCKET_TYPE:link` - `ozonesecure/test.sh` — `SCHEME:ofs`, `BUCKET_TYPE:bucket` - `ozonesecure-ha/test.sh` — `SCHEME:o3fs`, `BUCKET_TYPE:link` **Proposal:** Treat `AWS_CLI` the same way. Each existing full-`s3/` invocation runs **once**, with either `aws` or `aws2` passed as `-v AWS_CLI:...`. Both CLIs are covered **across CI legs**, not **within each leg**. 1. **Remove** `execute_s3_smoketests` from `hadoop-ozone/dist/src/main/compose/testlib.sh`. 2. **Replace** each `execute_s3_smoketests ...` call with a single `execute_robot_test ... -v AWS_CLI:<cli> ... s3`, using the assignment table below. 3. **EC legs:** Export `AWS_CLI=...` in `ozone/test-ec.sh` and `ozonesecure/test-ec.sh`. In `common/ec-test.sh`, use: ```bash execute_robot_test scm -v AWS_CLI:${AWS_CLI:-aws} -v BUCKET:erasure s3 ``` No new acceptance suites and no workflow changes. The number of full `s3/` invocations remains the same as on master before this PR. ### How to Split v1 vs v2 **Pattern A - One CLI per script (most legs)** | `#suite:` | Script | Proposed CLI | |---|---|---| | HA-unsecure | `ozone-ha/test-haproxy-s3g.sh` | **aws** | | secure | `ozonesecure/test.sh` | **aws** | | HA-secure | `ozonesecure-ha/test.sh` | **aws2** | | HA-secure | `ozonesecure-ha/test-haproxy-s3g.sh` | **aws** | | misc | `ozonesecure/test-vault.sh` | **aws** | **Pattern B - EC pair (unsecure vs secure)** | Script | Proposed configuration | |---|---| | `ozone/test-ec.sh` | `export AWS_CLI=aws` | | `ozonesecure/test-ec.sh` | `export AWS_CLI=aws2` | **Pattern C - HA-unsecure layout loop (split across existing runs, no extra invocations)** `ozone-ha/test.sh` already runs the full `s3/` suite three times: `OBJECT_STORE`, `LEGACY`, and `FILE_SYSTEM_OPTIMIZED`. The `no-bucket-type` tests are excluded after the first pass. **Proposal:** Add a `case` on layout to select the CLI: | Pass | `BUCKET_LAYOUT` | Proposed CLI | Notes | |---|---|---|---| | 1 | `OBJECT_STORE` | **aws2** | Full directory, including `no-bucket-type` tests | | 2 | `LEGACY` | **aws** | Excluded tags | | 3 | `FILE_SYSTEM_OPTIMIZED` | **aws2** | Excluded tags | **Expected totals:** 10 full `s3/` invocations — **6 × aws** and **4 × aws2**, versus 20 invocations if every current call site retained the double-run behavior. ### Why This Split? 1. **CI cost:** One CLI per invocation brings wall-clock time and total work back to **one pass per leg**, addressing the double-run issue on the critical path. 2. **Reuse existing dimensions:** The only job that already performs multiple full `s3/` runs is the HA-unsecure layout loop. Assigning `aws` to LEGACY and `aws2` to OBJECT_STORE and FILE_SYSTEM_OPTIMIZED splits v1/v2 **within that existing loop**, with zero extra Robot invocations. This is the closest equivalent to using different backend knobs instead of running everything twice. 3. **v2 on selected HA/secure paths:** `aws2` covers HA-unsecure OBJECT_STORE (full suite) and FILE_SYSTEM_OPTIMIZED, the HA-secure main S3 run, and secure EC. 4. **Default v1 elsewhere:** Legs that historically ran one full `s3/` suite with implicit `aws` remain on `aws` unless explicitly listed above. ### Trade-off Each Robot case runs under **one CLI on one backend configuration**. We do not run every test with both CLIs on the same cluster. Regressions that occur only with the other CLI on the same backend would require a different leg or a targeted test. **Does this partition look reasonable before I commit and push the changes?** I am happy to adjust the `aws` vs. `aws2` mapping, especially across HA layouts, secure-HA, and EC, if you prefer a different split. cc @adoroszlai @jojochuang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
