JoshRosen commented on PR #57710: URL: https://github.com/apache/spark/pull/57710#issuecomment-5169180106
I had Claude compare this PR's test durations to a sample of master branch CI runs: <img width="1100" height="612" alt="ci-shard-strip" src="https://github.com/user-attachments/assets/d15df4b6-8399-4f77-804d-bcc0235bedd3"; /> Its notes on methodology: > Methodology: Each row compares one CI job's duration in the patched run (orange diamond, n = 1) against the same job across ~40 recent post-merge build_main.yml runs on apache/spark master (blue dots, one per run), from GitHub Actions' job timestamps. Only full-workload runs are included (> 0.6× the job's max duration, since module detection trims test content per merge), jobs need n ≥ 8 comparable runs, and sub-10-minute jobs are excluded because fixed overhead (checkout, image caches) dominates their duration, making relative variance uninformatively large. z = standard deviations between the patched run and the master median; negative = faster, and |z| ≳ 2 exceeds normal run-to-run noise. Dots are translucent with slight deterministic jitter so dense clusters stay countable. The huge drop on "sql - other tests" seemed a bit suspicious to me; we double checked and confirmed that this _isn't_ due to test selection effects: > Restricted to the identical 897 test suites, summed per-test time dropped 22.8% (108.6 → 83.9 min), matching the job-level wall-clock delta exactly. So this appears to be a pretty significant overall wall clock time saving in CI, at least for some suites. --- As an aside, I think there's probably a lot of low- to medium-hanging fruit in profiling / analyzing the slowest individual test cases in general. I spotted a few pre-existing outlier-slow tests (~22 minutes for one test case, ~3 minutes for another) that are probably optimizable via scale tuning, better waiting / cross-thread coordination patterns (esp. in streaming test assertions), etc. An interesting project might be to write an aggregator of the master branch CI's JUnit XML files to look at both suite- and test-level timing + flakiness, then point agents at outliers for first-pass triage. There's probably more low-hanging fruit from local JFR profiling of "big suite of tiny test cases" scenarios, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
