morningman opened a new pull request, #4165: URL: https://github.com/apache/doris-website/pull/4165
## What The 2026-09-21 12:05 cron deploy ([run 35597560821](https://github.com/apache/doris-website/actions/runs/35597560821/job/106325798481)) spent **2h04m** in "Upload files to OSS" and then failed, so the asf-site deploy was skipped and the site stayed on the 08:43 build. From the step log: - The runner → OSS link was ~50x slower than usual for the whole two hours: ~2 successful PUTs/s (the 08:06 run did ~110/s), and the final ~9 MB `.actioninfo` PUT alone took 16 minutes (6 s normally). #4163 had changed the global CSS, so every HTML page (12,320 objects) had to be re-sent. - **6 of the 12,320 PUTs** hit the SDK's 60 s timeouts (`net/http: timeout awaiting response headers` ×4, `write tcp … i/o timeout` ×2). The action exits 1 on any failed upload and has no retry (the OSS SDK it embeds never reads its `RetryTimes`), so a 0.05% failure rate failed the step - after 99.95% of the upload had gone through and been recorded in the bucket's `.actioninfo` manifest. ## How 1. **Retry the upload step once** (`cron-deploy-website.yml`, `manual-deploy-website.yml`). The first attempt gets `continue-on-error: true`; a second identical step runs only when `steps.oss-upload.outcome == 'failure'`, and a failure there still fails the job. The action writes `.actioninfo` before exiting non-zero, so the retry only re-sends the objects that failed plus the manifest. In this incident that would have been 6 objects, and the job would have gone on to deploy asf-site at ~15:25 instead of stopping. 2. **Concurrency group for the cron workflow** (`cron-deploy-website`, `cancel-in-progress: false`). A slow run and the next 2-hourly trigger otherwise push the same objects through the same pipe and race for the manifest. The newer run queues instead; only the latest pending run is kept, which is what a deploy-master cron wants. The manual workflow deliberately gets no group: sharing one with the cron would let a cron trigger evict a human's pending manual run, and separate manual runs are rare and supervised. Not done on purpose: - **No `timeout-minutes` on the upload step.** The action writes its manifest only at the very end, so killing a slow upload discards the record of everything sent so far and the next run starts the whole upload again. A 30-minute cap would have left the site undeployable for as long as the slow period lasted, whereas the uncapped run plus one retry would have deployed. A genuine hang is not possible either: every request is bounded by the SDK's 30 s connect / 60 s read-write / 60 s header timeouts. The job keeps GitHub's default 6 h ceiling. ## Follow-ups outside this PR - If the evening-peak (UTC 13:00–15:00 = Beijing 21:00–23:00) slowdown recurs, switching the `ALIYUN_OSS_ENDPOINT` secret to the bucket's transfer-acceleration endpoint (`oss-accelerate.aliyuncs.com`, after enabling it on the bucket) moves the cross-border hop onto Alibaba's backbone. Secret-side change, extra cost. - `cron-generate-pdf.yml` / `manual-generate-pdf.yml` upload `build-pdf` with the action's default `incremental: true`, which would overwrite the shared root `.actioninfo` with a manifest containing only the PDFs and force the next website deploy to re-send all ~63k objects. It is latent today because "Generate PDF" has failed on every recent run, but worth `incremental: false` before anyone fixes the generator. ## Verified Both files parse (js-yaml); the retry step's `with:` block is byte-identical to the first attempt's; `if:` uses `outcome` (not `conclusion`, which `continue-on-error` reports as success). Not exercisable before merge: the workflows run on `schedule` / `workflow_dispatch` from master. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MvJw7xdszLCGBQdrvi7N9Z -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
