Wei-Chiu Chuang created HDDS-16651:
--------------------------------------
Summary: Retry K3s installer downloads and preserve installation
errors in Kubernetes CI
Key: HDDS-16651
URL: https://issues.apache.org/jira/browse/HDDS-16651
Project: Apache Ozone
Issue Type: Bug
Components: CI
Reporter: Wei-Chiu Chuang
The Kubernetes CI check can fail before any Ozone tests run when fetching the
K3s installer fails. The download is not retried, and the underlying error is
hidden by a subsequent chmod failure.
Observed on master commit ba05c2e3ad8151251f61732a00f98bccefe33f3b:
https://github.com/apache/ozone/actions/runs/36542831366/job/109327361915
{noformat}
2026-09-29T08:43:21.3475024Z ~/work/ozone/ozone/.dev-tools/k3s
~/work/ozone/ozone
2026-09-29T08:43:21.4148590Z chmod: cannot access '/etc/rancher/k3s/k3s.yaml':
No such file or directory
2026-09-29T08:43:21.4158779Z Failed to install k3s
{noformat}
In hadoop-ozone/dev-support/checks/install/k3s.sh, _install_k3s runs:
{code:bash}
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="${K3S_VERSION}" sh -
sudo chmod a+r $KUBECONFIG
{code}
The curl command has no retry option and suppresses error messages. Although
kubernetes.sh enables pipefail, the function does not explicitly return after a
failed pipeline. _do_install invokes the function via an if eval condition, so
execution continues to chmod; that error masks the initial failure.
The absence of installer output and the approximately 67 ms elapsed time
suggest the installer download failed. The exact network/HTTP cause cannot be
established from this log. The installer endpoint responded successfully during
investigation.
Suggested fix:
* Add bounded retries for transient installer download failures.
* Preserve curl error output (e.g. use -fsSL rather than -sfL).
* Explicitly return on download/install pipeline failure before attempting
chmod.
* Add shell coverage for a failed download and a retry that succeeds, including
verification that chmod is not attempted after installation failure.
The continuation to chmod after a failed download was reproduced with a mocked
curl returning exit code 22 under the existing conditional invocation pattern.
Duplicate check: searched unresolved HDDS issues for k3s, get.k3s.io, and
download combined with retry/kubernetes/installer. HDDS-5528 concerns volume
permissions with newer K3s versions and is unrelated.
Generated-by: Codex (GPT-6)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
