Siyao Meng created HDDS-16156:
---------------------------------
Summary: Flaky kubernetes check: s3g CrashLoopBackOff with "Access
denied: ozone_http_tmp_base_dir" (HTTP base dir created under non-writable CWD)
Key: HDDS-16156
URL: https://issues.apache.org/jira/browse/HDDS-16156
Project: Apache Ozone
Issue Type: Sub-task
Components: kubernetes, s3gateway
Reporter: Siyao Meng
h3. Symptom
In the {{kubernetes}} acceptance check, the {{s3g}} container
CrashLoopBackOffs: it exits immediately at startup, before serving anything, so
cluster bring-up for the {{ozone}} example (and others) fails ("Test execution
of ozone is FAILED"). Every {{s3g}} pod across all k8s examples in the run
(getting-started, ozone, ozone-dev, ozone-ha, minikube) fails identically;
{{scm}}/{{om}}/{{recon}}/{{httpfs}} start fine.
{code}
2026-08-11 22:11:26 INFO TracingUtil:87 - Initialized tracing service:
S3gateway (enabled=false, applicationAware=true)
Error: Access denied: ozone_http_tmp_base_dir3661902540861472713
{code}
(from {{kubernetes/.../s3g-0}} pod log)
h3. Root cause
{{HttpServer2.setHttpBaseDir()}} creates the HTTP temp base directory under the
process *current working directory* when {{ozone.http.basedir}} is unset:
{code}
// hadoop-hdds/framework/.../server/http/HttpServer2.java:1808-1815
File tmpMetaDir = Files.createTempDirectory(Paths.get(""),
"ozone_http_tmp_base_dir").toFile();
{code}
When the container's CWD is not writable, {{Files.createTempDirectory}} throws
{{AccessDeniedException}}, which {{GenericCli}} reports as {{"Access denied: "
+ message}} (GenericCli.java:157) and the process exits. Only {{s3g}} is
affected: it is stateless and has no metadata dir, so {{ozone.http.basedir}}
stays empty and it falls into this CWD path, whereas {{scm}}/{{om}}/{{recon}}
resolve {{ozone.http.basedir}} to their writable metadata dir and never enter
it (their logs never reference {{ozone_http_tmp_base_dir}}). The s3g
statefulset sets no {{workingDir}}, so the CWD is the image default. This makes
s3g startup fragile to the runtime environment's CWD writability, so the
{{kubernetes}} check fails intermittently (it is green on master for the same
code/image). The CWD-relative temp dir was introduced in HDDS-9483 (see also
HDDS-5239).
h3. Fix direction
Create the HTTP base dir under a guaranteed-writable location
({{java.io.tmpdir}}, or the service metadata dir) instead of the process CWD
({{Paths.get("")}}); alternatively set {{ozone.http.basedir}} for s3g. Also
consider surfacing the offending path in the error so this is not just "Access
denied".
h3. Notes
Observed on a PR build (run below) but *not* caused by that PR: the failing
code path predates it and runs during early s3g startup, before any SCM
interaction, so it is unrelated to the change under test. PR-build
logs/artifacts expire; the s3g pod log snippet above is the evidence.
- https://github.com/apache/ozone/actions/runs/31539983527/job/93943601739
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
